A Two-stage Method And Application Implementation For Image Generation From Sketch

Sketch-based image generation is an important issue that needs to be solved urgently in computer vision related tasks.For hand-drawn sketches,different users have huge differences in the expression of the same object,this requires the algorithm model to be able to learn the huge differences within the sketch category.Moreover,there have huge differences in scale and texture between sketches and images,the gap between features makes the training of sketch to image generation models more difficult.To fill the feature gap between sketches and images,this paper proposed a two-stage sketch to image generation method from sketch to edge map,and then from edge map to image,and apply the model to realize an image generation system based on multi-class sketches.In the first stage,our model learns the mapping of the sketch to the edge map based on generative adversarial networks,to realize the mapping from sketch to image size and structure,reduce the difficulty to generating images from sketches directly;In the second stage,the combination of the sketch and the generated edge map is used as input to provide stronger shape constraints for the training of generating real image,which realizes the mapping from image structure to texture.In addition,this paper introduces correlation learning among features,by using the combination of mutual information regularization constraint and loss function to understand the class difference between the sketch data,solve the problem of generating images from multi-class sketches and improve the learnability of the original model.For sketch-based real-time image generation system,we used virtual DOM technology to solve the problem of high-frequency data update caused by changes of strokes and lines in the process of drawing sketches,used Node.js to realize sketch data calculation and management,which can reduce the degree of coupling between the algorithm model and the system,and realize real-time interactive applications based on B/S architecture.The method proposed in this paper is compared with state-of-the-art sketch to image generation methods,which have 5.2%improvement in per-pixel accuracy.Especially in the generation tasks of horses,zebras and giraffes,the overall shape generation and detailed texture description have a better performance.In the end,by running the sketch to image generation system in the real scene,verified the model proposed in this paper is effective in generating images from multi-class sketches and have practical application value.