Research On Methods Of Cross-modality Face Synthesis

As smart phones and the Internet influencing the tens of different families, face images in different modalities are often encountered in specialized field and our daily life. Such as face images in photo and sketch style, visible light and nearinfrared style and so on. For example, we want to translate a face image from photo style to the corresponding face sketch image, in order to meet the practical needs of digital entertainment and professional tasks. Automatic cross-modality human face synthesis aims to convert face images between different styles. As an active yet challenging at present, it has great research significance. Currently many researchers have proposed a number of methods for this task, but there are some disadvantages in visual effects.In this thesis, we improve algorithms of cross-modality face synthesis to get the further improvement of experimental results at some extent. Firstly, we propose a two-step detail-enhanced face synthesis algorithm based on guided image filtering. The first step is the KNN-based method to select the most similar image patches for a test image patch and combine these patches together for approximating the test image patch. The second step is using guided image filtering on initial synthesis results with the input test image as guided image, in order to make up the sh ortage of lacking details. Through qualitative evaluation, it can simultaneously keep global features and enhance fine-scale details. Secondly, based on the above algorithm, we proposed the method called structured detail enhancement for cross-modality face synthesis. The main idea is that adopting different synthetic strategies for facial component and other facial part. Thus, it can effectively address the lack of losing shadow detail in facial components. Thirdly, because of the phenomenon of facial component's location bias for some test face images, the experimental results show that the reason is that in training set the facial components are not completely aligned in source modality and target modality. Therefore aligned training set can be adopted to improve the results. Meanwhile, we propose an improved landmark detection algorithm for face sketch image which can be used to align data set. Using aligned face photo image and sketch image pair can effectively tackle the problem of that the locations of generated facial components offset from the locations they should be in and the problem of that the outline are not clearly.The aligned training set can also be adopted in the methods based on convolutional neural networks, and generative loss and discriminative loss describe the loss function together. We fine-tune the pre-trained model using the aligned data set, and the experimental results show that some visual effect enhanced to some extent. In addition, the proposed three methods in this thesis were compared with current classical algorithm, concluding the comprehensive summarization on the visual effects, quantitative analysis and the operating speed. In the end, further works for this task were planned.