Real-Time Image And Video Abstraction

The art of abstraction reflects the abundant imagination of human beings and plays an important role in the development of human civilization. Nowadays, cartoon effects have been widely applied in a variety of areas including movie, television, game, network, advertisement, scientific illustration, and medical imaging. China has the biggest consumer market in the animation industry and the biggest online chatting group. The simplified or even exaggerate visual information after abstraction can often improve the human perception and plays as a communication bridge between artists and viewers. The image and video abstraction is an important technique to convert real photographs or recorded videos to non-photorealistic styles. Consequently the research on it has not only the theoretical significance but also the realistic application value. This paper mainly focuses on some key issues on the image and video abstraction, specifically, color-based photo style transfer, feature flow-based image and video abstraction, shape-simplifying image abstraction, perception-based progressive image and video abstraction, and GPU-based optimization algorithms.1. We propose a real-time color-based photo style transfer method. We first transform the input image from RGB color space to oRGB color space. The oRGB supports independent manipulation of the luminance, pure yellow-blue, and pure red-green color channels. Then the statistical color correction technique is performed on the two chrominance channels. We also apply the histogram matching on the luminance channel. We optionally add an abstraction effect on it to improve the visual perception. At last, the result image is produced by transforming the image back to the RGB color space. When transferring a target image's style to a video, since the target image is unchanged, we can compute its statistical information only once to reduce the processing time.2. We present a real-time feature flow-based image and video abstraction framework. First the input image is converted from RGB color space to CIE-Lab or oRGB color space. Next we iteratively construct a smooth, coherent, and feature-preserved edge tangent flow field using a bilateral filter. Using this feature flow, low contrast regions are smoothed with the flow-based bilateral filter, while high contrast regions are further strengthened with the flow-based difference-of-Gaussian filter. Then the soft luminance quantization is adopted to further enhance the cartoon-like effect with good temporal coherence. Lastly, the output image is generated by converting the result back to RGB space.3. We adopt a more elaborated visual perception model and introduce a real-time progressive image and video abstraction framework. In order to reduce both spatial and temporal video noises, we view time as the third dimension and apply a 3D bilateral filter on the input video first. Then we derive a smooth regions-of-interest function based on a visual saliency model, which effectively describes the human attention. Lastly, we use the regions-of-interest function to control our automatic progressive image and video abstraction.4. We design a real-time framework for shape-simplifying image abstraction, which simultaneously simplifies both the shape and color information in the input image. We iteratively apply the bilateral filter on edge tangent curves to obtain a smooth and coherent feature flow field, which indicates the salient feature directions of the image. Then we iteratively simplify and shrink the whole shape of the image using the feature-flow-constrained mean curvature flow. Finally, we protect important shape edges with the Shock filter. In order to intuitively control the abstraction level, we can iteratively and progressively apply the above filtering process. Since the color includes red, green, and blue channels, to obtain better color simplifying effects, we perform the abstraction on these three chrominance channels independently.5. All the above image and video abstraction algorithms proposed are designed to suit for the parallel processing characteristic of graphics hardware. Therefore, our methods are highly parallel, enabling real-time implementation on the programmable graphics processing unit. Additionally, our abstraction systems do not require any user interaction and even naive users can easily generate various vivid cartoon effects. At last, the automatic and real-time abstraction techniques make them convenient to process online images and videos.