Face Sketch Synthesis Based On Sparse Representation And Greedy Search

Face Sketch Synthesis, which models the complex mapping between photos and sketches by machine learning methods and employs learned mapping to synthesize corresponding sketches of input photos, is of great significance to digital entertainment and law enforcement. For example, when a crime happens, suppose that the police cannot obtain videos including the suspect due to the environment conditions and the lack of cameras, in these cases, the best substitutes are often sketch drawings based on the recollection from eyewitnesses or victims. After obtaining the sketch drawing, the police can first convert the photos from police mug-shut databases into corresponding sketches which form mug-shut sketch databases. Then the obtained sketch drawing will be applied to locate or narrow down potential suspects from the mug-shut sketch databases. In addition, with the development of social media, many young people hope to own characteristic images as portraits of their accounts, among which various stylistic sketches are good choices. What is more, face sketch synthesis can still be an important component of other computer vision tasks, such as face sketch aging etc.Existing face sketch synthesis methods can be roughly categorized into two groups: model-based methods and data-driven methods. We devote ourselves into data-driven methods in this dissertation. Aiming to overcome shortcomings of existing data-driven methods, such as restricting the range of test photos and depending on a large number of training photo-sketch pairs, the main contributions of this dissertation are summarized as follows:1. A photo-sketch pairs based face sketch synthesis method is proposed. Existing data-driven methods only consider local search strategy. This will lead to dissatisfactory synthesized results of certain test photos with some non-facial factors. In addition, the same geometric normalization step between testing set and training set, which restricts the size of test photos, is necessary to local search strategy. Considering this fact, we develop a photo-sketch pairs based face sketch synthesis method. The first step is to transform each image patch into sparse representation by utilizing sparse coding algorithm which can improve the robustness of the proposed method against interference. The second step is to utilize both the sparse coefficient values and the selection orders of dictionary atoms in sparse representations of training image patches to build a search tree which can improve the search accuracy and speed for nearest neighbors. The third step is to infer the final sketch by combining the prior information of test photos with graphical model. The first and second steps are called sparse representation based greedy search for short. Experiments demonstrate that the proposed method is faster and more advanced to synthesize non-facial factors than existing data-driven methods. Besides, the range of the test photo is free.2. A single photo-sketch pair based face sketch synthesis algorithm is proposed. Existing data-driven methods need a large number of photo-sketch pairs as the training set. However, the cost of numerous photo-sketch pairs acquisition is high which confines the practical applications of existing data-driven methods. Besides, in some extreme cases, there is only a single photo-sketch pair available. Considering above fact, we propose a single photo-sketch pair based face sketch synthesis method. The first step is to build a Gaussian image pyramid for the single training photo-sketch pair. This preprocessing can not only increase the number of training samples but also consider the symmetry structure information of human faces. The second step is to apply sparse representation based greedy search to obtain a rough initial synthesized sketch. Thus, the initial sketch overcomes the drawbacks which photo-sketch pairs based face sketch synthesis method improves. The third step is to form a new training set combining the pair of the test photo and its initial sketch with the original single photo-sketch pair. Then we explore cascaded regression strategy and graphical model to conduct final face sketch synthesis. Experiments demonstrate that the proposed method can achieve comparable performance than the state-of-the-art data-driven methods. In addition, the proposed method can also synthesize non-facial factors and the range of the test photos is free too.3. A target sketch based face sketch synthesis approach is proposed. Existing data-driven methods mostly depend on training photo-sketch pairs, whether large or a single, which restricts the generalization ability of previous methods to produce arbitrarily stylistic sketches. Considering above fact, we design a target sketch based face sketch synthesis method which only needs a target sketch as the training set. The first step is to apply the sparse representation based greedy search strategy to estimate an initial sketch for a test photo. The second step is to utilize multi-scale feature to search for candidate sketch patches. The third step is to exploit multi-feature-based optimization model to refine the obtained candidate sketch patches. The last step is to further enhance the quality of final synthesized sketches by a cascaded regression strategy. Experimental results validate that the performance of the proposed method is comparable with state-of-the-arts. What is more, the method can generate high quality stylistic sketches corresponding to any given test photos in these cases where the listed stylistic target sketches are taken as the training set. Above advantages improve the entertainment of the proposed method.4. A unified framework for face sketch synthesis is proposed. Existing data-driven methods utilize local position search strategy while above three methods utilize global range search strategy. In addition, a linear combination of multiple candidate image patches is exploited to obtain final sketch in these methods. This will result in smooth synthesized results. However, existing reconstruction strategies of high frequency are model-based approaches which cannot keep sketch style well. Focusing on above problems, a simply unified framework for face sketch synthesis is introduced. The first step is to divide the training set into initial training set and high frequency training set. The second step is to search for candidate image patches in the initial training set by combining both local position search strategy and global range search strategy which consider both position constraint and the similarity information among global range. Then the initial sketch is obtained by graphical model. The third step is to search for candidate image patches in the high frequency training set by combining both local position search strategy and global range search strategy too. Then the residue is estimated by graphical model. So the final sketch is generated by combining the obtained initial sketch and residue information. Experimental results validate that the proposed method can handle with both non-facial factors and face components. Furthermore, the high frequency details are richer.In summary, this dissertation introduces four face sketch synthesis methods based on sparse representation based greedy search to improve the practicability of face sketch synthesis from the way of reducing training samples gradually and considering search range from partial to full. Theoretical analyses and experiments show the superior performance over existing methods.