Coarse-to-Fine Face Sketch Synthesis With GAN And CNN Models

Heterogeneous image synthesis refers to the face images attained from various sources,such as face photos taken by cameras with different light variations,hand drawn sketch portraits by the artists,software based composite faces and infrared images taken by infrared devices.In recent years,it has drawn great attention to its most useful applications in digital entertainment and specially in law enforcement due to sketch based image synthesis and recognition.Face sketch synthesis mainly means to generate a sketch image by giving an input photo image,which basically models the manifold mapping relationship between the sketch-photo images by some synthesis methods and employs that learned mapping relationship to synthesize the corresponding sketch images of the input photo images.In an ideal manner,the synthesized sketch or photo image should be more preserved in appearance and also realistic as sketch/photo image,so that it will render the both fine perceptual quality and high sketch recognition accuracy rate of synthesized images.Due to these aspects,this thesis provides a holistic study on face sketch synthesis and recognition based on both conventional and deep learning studies.Firstly,this thesis targets the comprehensive review and comparative study of some typical face sketch synthesis methods.The experimental and analysis perceptions study on existing face sketch synthesis methods is non-trial before.Due to the synthesis process associated with the training models,these existing methods are characterized into two fundamental types:Data-driven methods and model-driven methods.According to face sketch synthesis process,the data-driven methods also called exemplar-based methods are generally based on four segments:patch representation,neighbor selection,weight computation and patch assembling.But model-driven methods directly learn the mapping relationship based on generative model from face photo to face sketch images.In this study,some typical methods based on various kinds of sketch synthesis models are investigated and it draws some promising conclusions based on both qualitative and quantitative evaluations of the synthe-sized images.Many existing sketch synthesis methods straightly learn the relationship between photo and sketch images,and it's relatively difficult for these methods to take the discrete specific details for the sketch synthesis process.The synthesized sketches generated by these traditional methods consistently manifest the coarse textures of face images,however the fine details of a few critical facial segments are exclusively lost.By addressing these problems,we propose a novel framework for face sketch synthesis by applying deep learning features.It consists of two consecutive preprocessing synthesis network models based on deep learning.During first synthesis process,we use GAN model to get the coarse estimation generated synthesized sketch images,but these synthesized images are highly contaminated with noise and distortion.Then,we use a CNN model as the secondary processing tool for generated synthesized images.As a result,it fairly reduces the noise and distortion of the synthesized images and we got the fine estimation synthesized images and took them as the residual images for the network.Given the coarse synthesized images and these fine residual images,a fully functional network model is designed to get the final synthesized sketches with the fine and critical details in contrast to traditional techniques.Multiple experiments and analysis illustrate that our proposed method has remarkable results on the face sketch synthesis tasks as compared to the traditional methods based on deep learning.The performance evaluation of synthesized sketch images generated by face sketch synthesis methods is summarized as computational and theoretical bases.Sketch image quality assessment and recognition are the key processes to evaluate the synthesized sketch based face images.Both the qualitative and quantitative evaluations methods are performed by comparing the synthesized images with the ground-truth sketch images.The structural similarity based various types of methods are presented to evaluate the performance evaluation of synthesized sketch images such as weighted structural similarity index,Gaussian approach and human specified methods.For face sketch synthesis methods,the SSIM metric is mostly employed to measure the effectiveness and efficiency of synthesized sketches.There are various face recognition methods to compute the recognition accuracy of synthesized images such as PCA,Eigenfaces and LDA based methods.For face sketch recognition,NLDA is generally applied to perform the sketch synthesis experiments by repeating the recognition process many times and find the best synthesized results against to variations between the reduced number of dimensions.Finally,a comprehensive summary of the thesis concludes the whole presented and proposed works for face sketch synthesis methods,with a few suggestions for the follow-up research and also indicates some directions for future research.