Manipulating Facial Appearance In Live Video Through Real-Time Intrinsic Decomposition

We propose a real-time method for intrinsic decomposition of face video,which allows for manipulating facial appearance in live video.Prior to runtime,the user applies edits to the intrinsic albedo and shading layers of a reference face image,which allows for a broad range of manipulations including virtual makeup and relighting.These edits are then transferred to the corresponding layers of the face in the video stream at runtime,in order to achieve the purpose of real-time manipulating facial appearance.The key challenge of this work is in extracting the albedo and shading layers in live video frames both accurately and efficiently.For this,we present an approach that takes advantage of human facial structure and recent advances in face landmark tracking,and uses our optimization framework based on color and structure priors to segment each frame in video stream into semantically meaningful regions,meanwhile utilizes optical flows to ensure that segments are temporal smoothing,and then according to segments,their decompositions are transferred to the corresponding pixels in the live video in an efficient and robust manner.The manipulations to the intrinsic layers are transferred to the video stream in a similar manner.We demonstrate our method on a variety of facial manipulations,such as reducing the appearance of wrinkles,changing the environment lighting,transferring color tone from another image,and changing hair color.Our experiments demonstrate this method to generate decomposition results better than those of state-of-the-art intrinsic video techniques,while achieving it in real time for the first time.