Image and Video Decomposition and Editing

Abstract

Image analysis and editing is an important topic in Computer Graphics community. My research has centered around understanding the colorful appearance of physical and digital paintings and images. My work focuses on decomposing images or videos into more editable data structure, called layers, to enable various efficient image and video re-editing operations. The creation of a painting, in the physical world or digitally, is a process that occurs over time. In the final painting, only a static arrangement of color is all that remains. The rich literature for interacting with brushstroke editing history cannot be used. To enable these interactions, we present a set of techniques to decompose a time lapse video of a painting into a sequence of translucent “stroke” images. We also present a pipeline for processing real-world videos of paintings capable of handling long-term occlusions, such as the painter’s hand and its shadow, color shifts, and noise. What if we only have a single image instead? Can we still extract similar layers? We proposed a technique to decompose an input image into layers successfully. In our decomposition, each layer represents a single-color coat of paint applied with varying opacity. We solve a constrained optimization to find translucent, spatially coherent opacity for each layer via RGB-space geometry. We demonstrate the utility of layers for recoloring and object insertion. Our most recent work introduces an extremely scalable and efficient yet simple palette-based image decomposition algorithm to extract additive mixing layers from an image, based on RGBXY-space geometry. This new geometric approach is orders of magnitude more efficient than previous work. For physical painting images, their colorful appearance is determined by the distribution of paint pigments across the canvas, which we model as a per-pixel mixture of a small number of pigments with multispectral absorption and scattering coefficients.We present an algorithm to efficiently recover this structure from an RGB physical painting image, yielding a plausible set of pigments. We demonstrate more paint-like edits such as tonal adjustments, selection masking, cut-copy-paste, recoloring, palette summarization, and edge enhancement in pigment space rather than in RGB space. Color carries a lot of semantical information. Thus, it is critical that any color-aware tool is able to model and preserve the high-level relationships between the main colors in an image, palette, theme or design. We present a palette-based framework for color composition for visual applications based on above layer decomposition techniques. We abstract relationships between palette colors as a compact set of axes describing harmonic templates over perceptually uniform color wheels. Our framework provides a basis for interactive color-aware operations such as color harmonization of images and videos. Because our approach to harmonization is palette-based, we are, for the first time, able to conduct perceptual studies evaluating preference for harmonized images and harmonized palettes in a controlled manner.