Inverse Render And Reconstruction Of Face Images With Weakly Supervised Learning

Most users apply image processing tools to edit facial images and videos for skin smoothing and acne removal before posting them on the Internet.After editing,face images tend to become unrealistic.This is due to partial loss of texture details and lighting information in face images during the editing processing.Using more sophisticated inverse rendering algorithms to edit face images can retain sufficient texture details and complete lighting information,making the edited face images look more realistic.At the same time,inverse rendering algorithms not only ensure consistency of the lighting information of face images before and after editing,but also modify the lighting information in the image according to the need of the task.For example,with inverse rendering,one can preserve sufficient texture details while modifying a darker image to a brighter one.This thesis work mainly studies reverse rendering of real face images.Our main tasks are as follows:1)Existing methods use synthetic face image data to fully supervise the Sf SNet network model based on deep learning and decompose a face image into three parts: the albedo map,normal map and lighting parameters before reconstructing the original face image.Since a real face image does not have the corresponding albedo map,normal map and lighting parameters,that is,there is no label that can be used for supervision,it is impossible to use the real face data to supervise the training network model.However,there is a domain gap between the real and synthetic face images.Models trained using synthetic face image data usually cannot be directly applied to real face image data.In order to solve this problem,we design an align-net structure based on the latest Sf SNet algorithm and proposes a weakly supervised training method.This method takes advantage of the identity consistency of the albedo and normal maps of different frames in the face video data,and weakly supervise the CNN model to solve the problem of domain separation between real face data and synthetic face data.The trained model can inversely render and reconstruct real face images effectively.2)The Lambertian lighting model used in the Sf SNet algorithm has its limitations: it uses spherical harmonic functions to represent lighting information,and is only applicable to Lambertian body.The Sf SNet algorithm only retains the first three low-order spherical harmonic coefficients while discarding the remaining high-order components.Furthermore,the real face does not belong to a complete Lambertian body,so the Sf SNet cannot handle some texture details and lighting information in real face image.To rectify this,we proposes an Illu Res-Sf S deep learning algorithm.On the premise of decomposing the albedo map,the normal map and the lighting parameters,the lighting residual map in the face image is further extracted.The lighting residual map contains texture details and lighting residual components that are ignored in the Sf SNet algorithm and Lambertian lighting model.The Illu Res-Sf S algorithm we propose can effectively improve the quality of the albedo map,normal map and reconstructed face image.This thesis comprehensively evaluates the performance of weakly supervised learning based on align-net and the Illu Res-Sf S algorithm both subjectively and objectively.Reported comparison results show that our weakly supervised learning based Illu Res-Sf S algorithm is superior to other inverse rendering algorithm for face images.