Freehand Sketch To Image Generation Based On Unsupervised Generative Adversarial Network

Sketching is one of the most natural and flexible ways for humans to express and convey information.In recent years,with the increasing popularity of touch screen devices such as smart phones and tablet computers,it has become one of the important human-computer interaction ways to express human visual needs by drawing sketches containing only simple lines on touch screens.Freehand sketch to image generation algorithm can realize the conversion of the sketch to an image.The color image corresponding to the freehand sketch can be generated by using the freehand sketch to image generation algorithm,and the algorithm can greatly reduce the workload of the painter.The algorithm has a wide range of application prospects,such as children’s painting education,clothing design,interior design,etc.,mature algorithm can even be used in criminal investigation.Due to its advantages of strong learning ability,wide coverage and end-to-end training,generative adversarial network(GAN)has been booming in the field of computer vision and image processing.Now GAN has become an important method and approach from freehand sketch to image generation.The lack of datasets is an inherent problem in the study of freehand sketch to image generation,especially the paired sketch-image datasets,and it’s expensive to produce.Besides,the freehand sketch lacks visual features such as color and texture.The high abstractness of freehand sketches results in a few quality problems in the generated image,such as shape deformation,colors or textures distortion.Aiming at the above problems,this paper carries out the research work based on freehand sketch to image generation.The innovation of the paper mainly includes the following two aspects.The paper proposes an algorithm of freehand sketch to image generation based on generative adversarial network.This algorithm solves the problem of freehand sketch to image generation from an unsupervised perspective.The core idea of the algorithm is unsupervised learning.That is,model adopts unsupervised training mode without paired freehand sketch-image datasets,which greatly decreases the requirement of the algorithm on datasets.To solve the problem of dataset shortage,we collect a new giraffe dataset,NewGiraffe.All samples of it are complete and clear.From the global and local aspects of the image generation,we propose different methods to improve the color and texture of the image.The proposed Image-Global attention mechanism(IGA)focuses on the global quality of the image generation.The focal frequency loss(FFL)is introduced to calculate the loss of each pixel of the real image and the generated image,to make the corresponding patch of the generated image and the real image as similar as possible.It can improve the local quality of the generated image.Image-Global attention mechanism and focal frequency loss play an important role in improving the color and texture of the generated image.Freehand sketches are extremely abstract,especially in shape,which will lead to unreasonable and unreal images in shape.Therefore,we propose a shape discriminator to constrain the shape of the generated image based on the unsupervised learning model.The algorithm proposed in this paper has made great progress in improving the quality of the generated image.A large number of experiments show that the proposed algorithm is superior to other sketch to image generation methods in three quantitative evaluation indexes:FID,SSIM and LPIPS,and our method can generate images with more complete and reasonable shapes,more realistic color and texture.