Research On Direction-aware Style Transfer

With the emergence of PRISMA and other AI art creation software,computer art creation has attracted great attention in the society.As an important technology to support this kind of software,Non-photorealistic Rendering involves many research area such as computer art,computer vision,computer graphics,digital image processing,etc.Non-photorealistic Rendering has a wide range of applications in animation,movies,games and other fields that need computer creation or computer-aided production.Traditional art creation industry is labor-intensive,which takes months or even years and a lot of manpower.With the development of artificial intelligence,how to apply computer in this field has attracted a lot of researchers’ attention.Example-based Rendering,which is also known as Style Transfer,is one of the hot research topics in the field of Non-photorealistic Rendering.It uses machine learning or statistical model to obtain the style characteristics from art image examples such as ink painting,oil painting,watercolor painting,embroidery,etc.,and then realizes the style transfer from style image to content image.The expressiveness of art mainly comes from color and texture,and texture of many artworks are composed of directional strokes aggregation,such as oil painting,embroidery,pencil painting,etc.How to express these directional textures is the premise of good style transfer effect.Some of the early works of Non-photorealistic Rendering put forward some solutions for the directional art forms.Although they have achieved certain results,they only use the lower level of style information,which makes the resulting stylized image less abstract.To solve this problem,Gatys et al.propose a method of style transfer using deep neural network technology,which has achieved great success.However,due to the ”black box” feature of neural network,this kind of method is difficult to control and display the texture details,so it can not deal with this kind of art form well.According to the research status and characteristics of the existing style transfer work,aiming at the art form with directional texture,this paper makes a systematic and in-depth study on several research difficulties,such as improving the artistic abstraction,enhancing the effect of detail texture and expressing the user’s intention.The innovative achievements mainly include the following aspects:1)Random-needle Embroidery is a unique form of embroidery in China.Compared with other art forms,Random-needle Embroidery mainly relies on the fine needlework texture produced by the aggregation of needlework threads for artistic expression.It has a relatively low degree of abstraction but requires a high level of texture detail expression.According to these characteristics of art forms,we proposes a texture synthesis style transfer method based on direction-aware sparse representation.According to the characteristics of embroidery texture,this method defines the rotatable primitives,and extracts the primitives from the style image according to the two characteristics of directionality and main direction.Then,this paper proposes a style primitives modeling method based on sparse representation to represent the texture of the primitives,and synthesizes the primitives according to the local features of the content image in the process of texture synthesis.Finally,we place primitives in the stylized image according to the direction field of the content image to synthesize the final stylized image.The experimental results show that our method can reproduce the texture details of stitches in stylized images and reduce the artificial traces effectively.2)In view of oil painting,watercolor and other art forms with directional texture and need a higher degree of artistic abstraction,we proposes a style transfer method based on directional field loss.First of all,this paper proposes a direction field loss network to supervise the direction field difference between stylized image and content image,so as to construct a new direction field loss to control the direction of strokes in stylized image;Then we combine the content style loss based on vgg-19 loss network with direction loss to construct a new optimization objective;finally,we generate stylized image by iterative online image optimization method.Experiments show that this method can better express direction texture in stylized image compared with the existing methods,and through simple interaction mechanism,users can control the direction of texture to better support user intention expression.3)Aiming at the art forms such as oil painting and watercolor painting,this paper proposes a direction-field based progressive style transfer method to further improve the quality of style transfer on the basis of maintaining stylized image generation efficiency.First of all,based on the idea of progressive image generation,we divides the process of style transfer into two stages: NST stage and texture enhancement stage.In the NST stage,in order to improve the efficiency of style transfer while preserving the resolution,an incremental generation network based on the loss of direction field is proposed to replace the off-line image optimization algorithm;Then,in the texture enhancement stage,a texture synthesis method based on the direction field is proposed to further improve the ability of texture detail expression on the basis of the stylized image obtained in the NST stage.A large number of experiments show that this method can further improve the texture details of style transfer results on the basis of maintaining enough artistic abstraction ability and certain generation efficiency.