Facial Intrinsic Image Decomposition And Its Application

Under the background of big data in the Internet era,people often use multimedia such as images and videos,instead of plain text,for storing and sharing information.This trend is evident from the evolution of social media from early blogs and Twitter to the recent Instagram and WeChat.The friends circle of WeChat supports short video sharing in its latest release.According to a study by the International Imaging Industry Association and a survey about the photo-sharing website Flickr,over 60 percent of shared or captured images and videos contain human faces.We believe this ratio will grow even higher because of the development of live streaming.In China,Meitu Xiuxiu,an application focusing on facial image enhancement,has served over 1 billion users and processes about 200 million images per day.Based on the aforementioned background,we study and explore human faces,which are typically the most important object in images or videos,including its intrinsic image layers,reflectance properties,3D geometry and environment illumination.We propose stable approaches for radiometric calibration,specular highlights separation and intrin-sic image decomposition on facial images by incorporating a statistical facial geometry prior as well as skin biological properties.Finally,a facial image enhancement applica-tion,virtual makeup,is presented based on these previous techniques.Our major work and contributions are listed as follows:1.We present a method for radiometric calibration of cameras from a single image that contains a human face.This technique takes advantage of a low-rank property that exists among certain skin albedo gradients because of the pigments within the skin.This property becomes distorted in images that are captured with a non-linear camera response function,and we perform radiometric calibration by solving for the inverse response function that best restores this low-rank property in an image.Although the approach makes use of the color properties of skin pigments,we show that this calibration is unaffected by the color of scene illumination or the sensitivities of the camera's color filters.Our experiments validate this approach on a variety of images containing human faces,and show that faces can provide an important source of calibration data in images where existing radiometric cali-bration techniques perform poorly.2.We present a method for removing specular highlight reflections in facial images that may contain varying illumination colors.This is accurately achieved through the use of physical and statistical properties of human skin and faces.We employ a melanin and hemoglobin based model to represent the diffuse color variations in facial skin,and utilize this model to constrain the highlight removal solution in a manner that is effective even for partially saturated pixels.The removal of high-lights is further facilitated through estimation of directionally variant illumination colors over the face,which is done by taking advantage of a statistically-based approximation of facial geometry.An important practical feature of the proposed method is that the skin color model is utilized in a way that does not require color calibration of the camera.Moreover,this approach does not require assumptions commonly needed in previous highlight removal techniques,such as uniform il-lumination color or piecewise-constant surface colors.We validate this technique through comparisons to existing methods for removing highlights as well as to works for estimating illumination colors.3.We present a method for decomposing a single face photograph into its intrinsic image components.Although current single-image intrinsic image methods are able to obtain an approximate decomposition,image operations involving the hu-man face require greater accuracy since slight errors can lead to visually disturbing results.To improve decomposition for faces,we propose to utilize human face pri-ors as constraints for intrinsic image estimation.These priors include statistics on skin reflectance and facial geometry.We also make use of a physically-based model of skin translucency to heighten accuracy,as well as to further decom-pose the reflectance image into a diffuse and a specular component.With the use of priors and a skin reflectance model for human faces,our method is able to achieve appreciable improvements in intrinsic image decomposition over more generic techniques.4.We present a physically-based approach for simulating makeup in face images.The key idea is to decompose the face image into intrinsic image layers-namely albedo,diffuse shading,and specular highlights-which are each differently af-fected by cosmetics,and then manipulate each layer according to corresponding models of reflectance.Accurate intrinsic image decompositions for faces are ob-tained with the help of human face priors,including statistics on skin reflectance and facial geometry.The intrinsic image layers are then transformed in appearance according to measured optical properties of cosmetics and proposed adaptations of physically-based reflectance models.With this approach,realistic results are generated in a manner that preserves the personal appearance features and lighting conditions of the target face while not requiring detailed geometric and reflectance measurements.We demonstrate this technique on various forms of cosmetics in-cluding foundation,blush,lipstick,and eye shadow.Results on both images and videos exhibit a close approximation to ground truth and compare favorably to existing techniques.