Image Style Transfer Based On Semantic Information

Painting is one of the most important visual forms of artistic creation.The artistic style of a painting includes a series of image information,such as color,texture and structure,which can reflect the artist's background and personal features.It is also the direct embodiment of various genres.Drawing a painting in particular style usually requires a well-trained artist and heavy time consumption.With the continuous development of computer technology,digital paintings can be excellently created with image processing and computer vision methods.By modelling the mathematics or statistics of the artistic style of a painting,image style transfer algorithm can automatically transform any image into specific artistic styles,in the meanwhile retaining its origin semantic content and structure.Image style transfer is widely applied in image editing,image creative design and other applications,which has important research significance and application value in social media,digital entertainment and other fields.Image style transfer has become one of the focus research areas of computer vision and image processing.Image style transfer focuses on the representation and processing of the artistic style(such as image color features,image texture features)and the content information(such as image structure,image semantics).The difficulty lies in how to establish a mathematical or statistical model for artistic style,and how to maintain the consistency of semantic content and structural information in style transfer.This dissertation investigates the application of semantic information in image style transfer based on machine learning and deep learning theory,and proposed a series of image style transfer methods with real photos as content images and artworks as style images.The main contributions of this dissertation are summarized as follows:1.A two-tone portrait synthesis method based on semantic segmentation is proposed.Existing image processing-based style transfer methods treat entire image as processing unit and would easily cause noise or lose some important details at edge area;while exemplebased methods divide image into patches,which destroys the structural integrity of the face,and finally leads to poor synthesis quality with artifacts in results.To tackle the aforementioned problems,a coarse-to-fine semantic segmentation is use to extract different semantic regions of face photos.Then different style transfer methods are applied for different semantic regions,which preserves the internal structural integrity and combines the advantages of different style transfer methods.In addition,global structure model is used to jointly model the component of different regions to balance the trade-off between content preservation and style resemblance,so as to realize black-and-white portrait synthesis.Experimental results demonstrate that this method has advantages and improvements over existing methods in synthesis quality of portraits,the preservation of facial structural,and the style resemblance with the example portraits.2.A semantic style transfer method based on dual-attention style embedding network is proposed.In style transfer,users usually pay more attention to the semantic alignment between style transfer results and style images.However,existing methods tend to transfer style patterns globally without considering semantic alignment.In addition,current semantic style transfer methods suffer from iterative optimization procedure,which is prone to be slow due to large computational cost,which fail to meet the real-time requirement of style transfer in social entertainment applications.In order to solve this problem,image semantic label information is introduced by semantic segmentation.A style embedding network based on semantic labels is designed to jointly model semantic correspondence with semantic attention mechanism and feature correlation with style attention mechanism to embed style representations.Experimental results demonstrate that compared with existing methods,this method can achieve style transfer with semantic alignment.The end-to-end network structure can improve the run time efficiency to achieve real-time semantic style transfer.3.An image style transfer method based on collection representation space and semantic guided reconstruction is proposed.Existing image style transfer methods based on deep learning have made decent progress with transferring the style of single image.However,visual statistics from one image cannot reflect the full scope of an artist or genre.In addition,previous work holistically constrains the content preservation between content image and style transfer results,but tend to lose local structural information,which would result in poor structure integrity,thus deteriorating the comprehensibility of generated image.In order to solve this problem,an encoding-decoding network is proposed to establish the representation space that can reflect the style of artist or genre,and learn the overall collection style through self-encoding reconstruction.In the meanwhile,semantic information is used as guidance to reconstruct the target representation of the input image for better local content preservation in style transfer process.Experimental results demonstrate that compared with existing methods,this method can capture collection styles and preserve local structural information,and therefore improve the visual quality of generated results.4.An image collection style transfer method based on dual-consistency loss is proposed.In the task of cross-domain style transfer,in order to translate images from content domain to style domain,the mapping between photos and artworks need to be learned through training.Existing image style transfer methods based on cross domain reconstruction only learns the mapping between content image and style image on pixel level,which ignored the relation between the style and subject of artworks.Thus the model fails to mimic artists' design mechanism,and result in the lack of diversity.In order to solve this problem,a semanticrelated photo-artwork dataset is constructed.A dual-consistency loss is proposed to train the generative adversarial network and supervise the style consistency and semantic consistency.In this way,the semantic correspondence between photo domain and art works is obtained.The experimental results demonstrate that compared with existing methods,this method can imitate artist's design mechanism and generate different style transfer results for different content,which improves the diversity and semantic correspondence of style transfer results while maintaining the structural information of content image.