Research On The Style Transfer Of Landscape Paintings Based On Deep Learning

The so-called image style transfer is to use the algorithm to learn the style of famous art paintings,and then transfer these style to other pictures.Since Gatys proposed to apply neural network to the field of style transfer in 2015,this method has rapidly become the mainstream method in this field.Although the current style transfer projects are widely used,they are mostly aimed at the style transfer of famous Western oil paintings.The most common style images are drawn by Picasso,Van Gogh,Cezanne and other Western oil paintings masters.However,there are few traditional Chinese paintings transfer projects.Traditional Chinese paintings are the artistic crystallization of Chinese traditional culture.They are treasure of Chinese civilization.The research on the style transfer of traditional Chinese paintings has an important application in both academic and industrial fields.This thesis will improve the cycle-consistent adversarial network(CycleGAN)to complete the style transfer of traditional Chinese landscape paintings.And to put forward solutions to some specific problems in the actual experiment.The details are as follows:(1)According to the style,landscape paintings can be divided into ink landscape paintings,green landscape paintings,splendor landscape paintings etc.Due to the fact that few people are interested in the style transfer of Chinese landscape paintings,This thesis only found the data of Chinese ink paintings in COCO data set.For the training of ink paintings,this thesis uses the existing shared data set to complete.But for green landscape paintings and splendor landscape paintings,this thesis can only make data set by ourselves.This data set establishment is based on tensorflow framework and apply two image processing libraries in Python:OS and PIL.This thesis establishes the data set through tfrecord.Tfrecords is a binary file,which can make good use of memory,is easy to copy and move,and does not need a separate label file.In addition,this thesis uses lableme to annotate the content image manually,which makes preparation for the following step of image semantic segmentation of local style transfer.This thesis regards the style transfer based on CycleGAN as baseline.The second chapter introduces the basic structure and principle of CycleGAN,and uses Cy-cleGAN to complete the style transfer of oil paintings and landscape paintings.(2)The third chapter mainly improves the traditional CycleGAN's style function and gener-ator structure.The loss function of CycleGAN is the cycle consistent loss function.This kind of loss function can not measure the structural similarity of pictures.In this chapter,MS-SSIM(multi scale structural similarity index)loss is added to the original loss function.SSIM is an index to measure the similarity of two images.MS-SSIM is an extension of SSIM.It mainly uses iterative method to filter and desample the image,and calculates SSIM index on each scale and synthesizes the results.The new loss function of MS-SSIM is divided into three parts:confrontation loss,feature matching loss and new cycle consistent loss.The experimental results show that the images generated by MS-SSIM-CycleGAN have a significant improvement in geometric characteristics and color brightness.However,when using the MS-SSIM-CycleGAN to generate images,there are often some checkerboard lat-tice shadows,that is chessboard effect.In this chapter,sub-pixel convolution layer is used instead of deconvolution layer in the upper sampling process of the generator.Experimental results show that the chessboard effect is effectively suppressed by introducing sub-pixel convolution layer.(3)This thesis Completes three special style transfer:the first one is local style transfer.As the name implies,local style transfer is to transfer artistic style to a part of content image.In order to complete this specific transfer,this thesis must firstly separates the foreground and background of the content image.So this chapter introduces U-net image segmentation network to complete this process.U-net consists of two parts:the first is feature extraction part,the second part is the upper sampling part,each time the upper sampling will be fused with the feature extraction part.After the separation of foreground and background,the results of partial style transfer can be obtained by stylizing the foreground content image;The second one is multi style transfer.The so-called multi style transfer is to transfer a variety of artistic styles to the same content image at the same time.This kind of transfer can be completed with a little modification based on the experiments of Gatys et al.The principle is that replacing traditional style loss function by the weighted sum of loss functions of multiple style images;The last one is dynamic image and video stylization.Dynamic image or video stylization should not only consider the artistic effect after stylization,but also ensure the time-domain consistency of video.In this chapter,a real-time video style transfer model based on the feedforward convolution neural network is combined with the deepflow optical flow method.The model of style transfer in this experiment consists of two parts:stylization network and loss network.Stylization network takes a frame of picture as input and produces corresponding stylized output.Loss network can extract the features of output frame of stylized network and calculate the space loss,which in turn is used to train stylization network.Space loss is the weighted sum of content loss and style loss,through which we can evaluate the quality of image transfer in space.In this experiment,deepflow is used to calculate the optical flow.(4)Using PyQt library in python to build an interactive design interface based on neural network,the various image transfer of this thesis are fully realized.In this human-computer interface,convolutional neural network can be trained directly,so that even people who do not know deep learning can complete the image style transfer.(5)The content of this thesis is summarized,and the future of image style transfer is prospected.