Shadow-and-Pose Robust Makeup Transfer

In daily life,suitable and perfect makeup will bring people with facial attractiveness and self-confidence.Without professional recommendation,it is timeconsuming to combine a satisfactory makeup style among thousands of cosmetics with different brands,colors,and uses.Digital facial makeup application can present reference makeup on the face virtually according to user’s demands,that is what you see is what you get.In this thesis,we conduct a research named makeup transfer which is the key technology in digital facial makeup application.Makeup transfer aims to the transfer the facial makeup style from any reference image to the source image with presenting a real,clear and accurate makeup effect,while preserving the identity of the source face.In the field of facial makeup transfer,many researchers have made many great contributions,with which in traditional image processing methods based on facial structural assumptions and physical simulations,and image generation methods developed with deep learning.Good results can be generated when the faces in images are frontal and standard,but it is still a challenge in many real-world scenarios such as when shadows and occlusions appear or when the faces with large pose and expression differences.In order to address the makeup transfer task which different makeup color exist before and after makeup transfer and inaccurate position correspondence between source images and reference images,we propose a shadow-and-robust makeup transfer method,main research contents are illustrated as follow:(1)For the problem that shadow disturbance in the reference images can interfere with the reference makeup color and lead to inaccurate makeup color after makeup transfer,a shadow manipulation module based on attention mechanism is proposed to manipulate the shadows appearing in the image while ensuring that the color under the shadow is not changed.The shadow manipulation module in this thesis includes two submodule,which are foreign shadow manipulation and facial shadow manipulation.Foreign shadows are mainly caused by irregular external objects,such as hats or other occlusions,while facial shadows are caused by face parts,such as nose and cheeks,and the general shadow location is on one side of the face.The main architecture of the foreign shadow manipulation is a two-channel hierarchical aggregation network,which is used to predict the extent and boundary of the shadow mask and to separate out the face under the shadow.The main architecture of the facial shadow manipulation is the symmetric attention module,which uses the prior knowledge of face symmetry and covers the lower quality region with lower confidence causing by facial shadow or black spots with symmetrical higher quality region with higher confidence,in order to obtain a makeup vector without facial shadow disturbance.The experiment results show that the shadow manipulation module can better separate the shadows without changing the face identity attributes or makeup color under the shadows,and the predicted images can be used in the next stage of the makeup transfer task,and the ablation experiments in Chapter 3 of the makeup transfer algorithm prove the effectiveness of the shadow manipulation module.(2)To address the problem that the faces with large pose and expression differences of the source and reference images leads to inaccurate correspondence positions in the process of makeup transfer,we propose a face component correspondence module for aligning the three important parts of the face including lips,eyes and cheeks,and transferring the makeup to the corresponding areas accurately.The face component correspondence module learns the attention matrix of the part correspondence through the feature vectors of the source image and the reference image,transforms the makeup matrix extracted from the reference image feature vectors to the source image through the attention matrix,achieves the accurate generated results with reference makeup for post robustness.The face makeup transfer algorithm for shadow and pose robustness in the thesis mainly includes three steps:firstly,the source image and the reference image are expanded into (1 space to obtain(1 position map and (1 texture map,(1 position map includes key geometric parameters such as face shape,and (1 texture map includes parameters such as face texture,makeup color and distribution,and then the (1 texture map of the reference image is passed through pre-trained foreign shadow manipulation module to remove foreign shadow,the facial shadow manipulation module reduce facial shadow disturbance,the (1 texture map of the source image and the reference image without shadow disturbance are fed to the face component correspondence module to align the important parts of the face and to transfer makeup,finally,the obtained (1 texture map is mapped to the (1 position map of the source image through the rendering process to output the generated makeup image.In summary,the algorithm is proposed in the thesis of shadow and pose robust facial makeup transfer,based on the shadow manipulation module in the two-channel hierarchical aggregation network and symmetric attention module which can reduce the interference of shadows on makeup,based on the face component correspondence module to align the source image and the reference image of the face position,so as to transfer the reference makeup to the correct source image position,to complete the whole task of makeup transfer.Qualitative and quantitative experimental shows,compared with existing makeup transfer algorithms,the algorithm in this thesis can transfer makeup better in scenes with shadow disturbances and different pose,while ensuring that the person identity attributes of the source image are not changed,and has better robustness in the makeup transfer task.