Foreground segmentation in images and video: Methods, systems and applications

Separating foreground objects from natural images and video plays an important role in image and video editing tasks. Despite extensive study in the last two decades, this problem still remains challenging. In particular, extracting a foreground object from the background in a static image involves determining both full and partial pixel coverage, also known as extracting a matte, which is a severely under-constrained problem. Segmenting spatio-temporal video objects from a video sequence is even harder since extracted foregrounds on adjacent frames must be both spatially and temporally coherent. Previous approaches for foreground extraction usually require a large amount of user input and still suffer from inaccurate results and low computational efficiency.; This thesis demonstrates efficient foreground extraction methods and systems by combining advanced computational algorithms with novel user interfaces. Our systems are capable of extracting high quality foreground objects from images and video with a limited amount of user input such as a few paint strokes of the mouse. We also demonstrate a variety of applications with the extracted foreground objects.; Specifically, for still images, we develop a novel Robust Matting algorithm, which is capable of generating high quality alpha mattes for complex images in a robust way. Centered around this algorithm we build Soft Scissors, the first interactive tool for extracting high quality mattes in realtime. We also propose a compositional matting algorithm which combines matting and compositing into a single optimization process. Quantitative and objective evaluations demonstrate that these systems outperform previous approaches in both accuracy and efficiency.; For motion pictures, we propose an interactive Video Cutout system which extracts spatio-temporal coherent foreground objects from video sequences through a novel 3D painting user interface. As an application we develop a Video Tooning system which can stylize the extracted video objects with a variety of cartoon styles. We also propose a cartoon animation filter which automatically exaggerates and stylizes the motion of the extracted video objects.