The Research On Simplifying Sketches With Perceptual Loss Based On Convolutional Neural Network

Sketching is usually the first step in drawing.A tedious and labor-intensive cleaning up or simplification process is usually followed to yield a neat drawing.To automate this simplification process,several methods have been proposed.Early works deals with vector input due to the complication in handling raster input and amazing results had been generated.There are attempts to convert a raster drawing to a vector form,but it could be very unstable due to the noises and lighting.Moreover,raster inputs are difficult to directly separate lines to get the relationship among them in comparison with vector sketches.With the advancement of deep learning,it is popular to train convolutional neural networks for image transformation tasks including sketch simplification.Direct simplification on raster sketch has been effectively demonstrated by Simo-Serra et al.However,the original DCNN-based sketch simplification is prone to blurriness as it tends to pixel-wisely reduce the training loss for candidate solutions.Generative adversarial network(GAN)is further adopted to avoid the blurriness,as the discriminator network randomly picks the most likely one from the candidate solutions,instead of averaging them.This can be explained by the fact that the discriminator network classifies the generated output as a simplified sketch or not,with less attention on the overall semantic structure.As a result,they tend to over-emphasize on the local structure and preserves many strokes that are less semantically important in a global scale.In summary,it is still challenging to simplify the extremely rough sketches.What we really want to achieve is preserving the details of the small scale at the same time suppressing the large scale of the clutter,while the current methods are not enough to achieve such a simplification similar to the human judgment with a lack of semantic understanding,.In this paper,a training framework combined with human judgment is proposed.The resultant simplification can preserve semantically important global structure as well as fine details,and effectively suppress blurriness.The following are our main contributions:1)The pretrained VGG16 network is used to measure the difference of line drawings simulating human judgments.It extracts high-level features that are highly related to human perception,and can obtain a line drawing that fits human perception simplification.By adopting Generative Adversarial Network,the trained network can generate clear and simplified results that are consistent with the characteristics of the real line drawings.2)A dataset is prepared,which can effectively simplify the various tough sketches in real life,such as scanning and photographing images,by simulating the illumination and shadow conditions.We also use semi supervised learning to make full use of labeled and unlabeled data and increase the variety of data through data augmentation to cover a large number of possible real sketches.3)To evaluate our method,we compare our method to state-of-the-art methods,through multiple experiments including visual comparison and intensive user study.Through user study,we evaluate from three aspects: overall aesthetics,tidiness and content conformity.The results show that our simplification network outperforms other networks in dealing with real sketches.