Cross-Domain Reconstruction Of Face Images Via Knowledge Representation And Transfer

Face image is the most common information carrier in the field of computer vision,carrying rich identity information.With the continuous development of face image acquisition technology,the modalities of face image are more abundant.For example,the visible face image taken by camera,the face sketch drawn by the painter,the cartoon face image made by computer,the near-infrared face image taken by the near-infrared imager and the thermal infrared face image taken by the thermal infrared imager are all the different modalities of face image.Face images taken by different modalities belong to different image domains.The information between different face image domains of the same target has both commonality and individuality,which provides a rich representation of different perspectives of the target.Cross domain face image reconstruction,aiming to reconstruct the face image of the target in one face image domain into the face image of other face image domains,can not only enrich the face information representation of the same target in different face image domains,but also reduce the representation differences of the same target in different face image domains,which has important research significance and plays important role in public security,social entertainment and other fields.Cross-domain face image reconstruction focuses on the representation and processing of common face information(such as face identity information,spatial structure information)and domain-specific face information(such as face modality information,face texture information)of the same target in different face image domains.The difficulty lies in how to transform domain-specific face information between different face image domains without losing common face information.The rise and development of deep learning has brought great vitality to this field.This thesis investigates the application of deep knowledge representation and transfer in cross-domain face image reconstruction based on deep learning theory,and proposes a series of cross-domain face image reconstruction methods.The main contributions of this dissertation are summarized as follows:1.A cross domain face image reconstruction method based on deep feature representation and probabilistic graphical model is proposed.Existing exemplar based cross domain face image reconstruction methods usually use pixel intensities as the features of image patches,or use some artificial feature extraction operators to extract the features of image patches.However,these low-level features have poor robustness and are easy to be interfered by noise such as illumination variation,clutter background,which eventually leads to poor quality of reconstruction results.To tackle this problem,deep feature representation extracted via deep convolutional neural networks is used as feature of face image patches.In addition,probabilistic graphical model is utilized to jointly model the channel weight of deep feature representation and the reconstruction weight of candidate image patches,and the model is optimized by an alternative optimization strategy,so as to realize the cross domain face image reconstruction robust to environmental noise.Experimental results demonstrate that this method has advantages and improvements over existing methods in the reconstruction quality of face image,face structure information preservation,face identity information preservation,and robustness to environmental noise such as illumination variation and clutter background.2.Existing deep learning based cross domain face image reconstruction methods have made some progress in the reconstruction speed and the realism of the reconstructed image,but tend to loss face identity information,which appears as a deviation between the face content of the reconstructed face image in the target image domain and the face image in the source image domain,and a decrease of the face recognition rate.To tackle this problem,the collaborative information of two dual reconstruction networks are collaboratively transferred to obtain the intermediate latent domain,so that the two dual reconstruction processes pass through the learned latent domain.By imposing constraints on the latent domain,the trained dual networks are more symmetrical and can preserve more common information.Experimental results demonstrate that compared with existing methods,this method can learn more symmetrical reconstruction mapping,therefore preserve more face content and identity information while reconstruct realistic face image.3.A cross domain face image reconstruction method based on knowledge transfer is proposed.In the task of cross domain face image reconstruction,in order to reconstruct the image between two image domains,a large amount of training data is needed to learn reasonable mappings.The main problem of existing deep learning based cross domain face image reconstruction methods is that the cross domain face image training samples are not enough,and the model can not fully learn the mappings.To tackle this problem,a deep knowledge transfer framework is proposed,which can guide the training process of cross domain face image reconstruction tasks by transferring the knowledge from other tasks pre-trained on sufficient training data and related to cross domain face image reconstruction tasks,so as to obtain more reasonable mappings.In addition,by transferring the knowledge of two dual cross domain face image reconstruction tasks mutually,the supervision effect of knowledge transfer is further improved.Experimental results demonstrate that compared with existing methods,this method can obtain more reasonable reconstruction mappings under insufficient training data,so as to reconstruct face image with higher quality.4.A cross domain face image reconstruction method based on facial label information transfer is proposed.The key of cross domain face image reconstruction task is to reconstruct the domain-specific information of different image domains without losing face identity and structure information.The main problem of existing deep learning based cross domain face image reconstruction methods is that the encoding-decoding structure of deep convolutional neural networks is tend to lose the structural information of the face image,resulting in the loss of the identity information of the reconstructed face image and the deterioration of the visual performance.To tackle this problem,facial label information is introduced through face parsing model and transferred to specifically designed cross domain face image generative adversarial networks to maintain structural information and obtain more reasonable mappings.Experimental results demonstrate that this method can improve the visual performance of reconstructed face image while maintaining reasonable face structural information.5.An application framework of cross domain face image reconstruction is proposed.The different cross domain face image reconstruction methods proposed in this dissertation are aimed at different specific problem,but in practical application,multiple problems may exist simultaneously.Therefore,an application framework of cross domain face image reconstruction is proposed,which integrates the cross domain face image reconstruction methods proposed for different problems,so as to apply them to the actual cross domain face image reconstruction tasks.Experimental results show that the proposed application framework has achieved satisfactory reconstruction results in the applications of face aging,face makeup,face artistic portrait generation,which demonstrates the effectiveness of the proposed cross domain face image reconstruction application framework in practical application.