Exemplar-based Texture Synthesis And Its Applications Research

Texture synthesis is one of the most fundamental techniques in computer graphics for texture mapping,which is also an important technique for image processing.This technique can be applied to image processing fields such as image inpainting and image stylization generation.The fundamental goal of example based texture synthesis is to generate a texture that faithfully present main visual characteristics of the exemplar,yet without other unnatural looking artifacts.Traditional methods have made remarkable progress towards this goal.Recently,deep learning becomes a popular method used in image processing.Besides,there are many extended applications.However,no matter the traditional method or the lately CNN method which is used for texture synthesis,none of them can deal well with the synthesis problem for large scale texture without artifacts.This thesis first surveys the existing state of artwork for texture synthesis and theory of deep learning.Based on this survey,the problems existing in the current texture synthesis method are summarized.In summary,there are three big problems.First,traditional methods are not universally applicable to all types of textures.Second,how to obta in the global optimal solution rather than the local optimal solution.Thirdly,how to obtain more features of exemplar for texture synthesis.In view of these problems,some novel ideas of texture synthesis are presented.Besides,the novel texture synthesis theory is further used in image processing.In specific,the main four contributions are as follows.1)In the first part,a new texture synthesis framework is proposed to solve the problems in traditional methods.We give priority to save local texture containing more information such as lines.Principal curvature is utilized to detect the existence of local area lines.Moreover,we do not need an extra channel.Meanwhile,in order to fully use local texture information,we make a diamond searching with decreasing radius in search step after propagation step.Besides,in noisy large-scale regular texture,if there is large-scale motif in texture,which is a difficult problem for texture synthesis.In this thesis,the motif first be extracted from the exemplar based on Scale-invariant features(SIFT).Then the motif will be take as the basic unit synthesized by patch based texture synthesis method.The experiment results show that the proposed method achieves significant improvements.2)In the second part,we present a new method for synthesizing a transition region between two source textures,such that inconsistent texture and structural properties all change gradually from one source to the other.We first extract the convolutional neural network(CNN)features of two source textures and one initial target texture at convolution and pooling layers.We set the distortion function to be the square of the difference between feature maps of source texture and target texture.In addition,we based on feature maps compute the Gram matrix that is the inner product between feature maps in each layer.We set the second distortion function to be the square of the difference between Gram matrix of second source texture and target texture.Our target funct ion is the weighted sum of two distortion functions we described before.We use Large-scale bound-constrained optimization method to optimize the target function and get the ultimate result.The model provides a new tool to generate a transition region between two source textures by CNN.Our method is robust to various types of images tested.3)In the third part,we use recurrent neural network(RNN)of long-shot time memory(LSTM)to generate a regular texture based on exemplar.The unnatural look ing artifacts can be generated because previous works cannot well catch the regularity for patterned texture.LSTM network can effectively handle long-range dependencies,which are central to object and scene understanding.We process the image pixel by pixel and allow the pixel to be processed in the LSTM end to end.Additionally,this structure ensures that the signals are well propagated in the space,and we take the regular texture as a periodic signal.Then,the number of hidden units for LSTM is set to be the length of signal period.We generate the synthesized texture in any size we wanted,which is also an improvement for texture synthesis to use deep learning technique.4)In the fourth part,the theory of texture synthesis based on CNN is applied to the image inpainting for object removal and image style transfer.CNN feature maps based inpainting method can effectually repair the large area missed image.Conditional Random Fields as Recurrent Neural Networks(CRF-RNN)is used to segment the target in semantic,which can avoid the trouble of mask or artificial pre-processing for object segmentation.In inpainting part,a new method for inpainting the missing region is proposed.Besides,the representation features are calculated from the CNN feature maps of the ne ighbor regions of the missing region.Then,L-BFGS is used to synthesize the missing region based on the CNN representation features of similarity neighbor regions.Compared to the traditional algorithm our method make obviously improved in peak PSNR,SSIM and the subjective quality.In addition,we propose a new method to produce pencil drawing from natural image.The synthesized result cannot only generate pencil sketch drawing but also can save the color tone of natural image and the drawing style is flexible.The sketch and style are learned from the edge of original natural image and one pencil image exemplar of artist' work.They are accomplished through using the CNN feature maps of a natural image and an exemplar pencil drawing style image.L-BFGS is applied to synthesize the new pencil sketch whose style is similar to the exemplar pencil sketch.Experimental results demonstrate that our method is better than conventional method in clarity and color tone.Besides,our method is also flexible in drawing style.