Research On Sketch Generation And Segmentation Methods Based On Auto Encoder Framework

With the development and progress of science and technology,more and more smart devices appear around people.A large part of these smart devices are devices with touch screens,including tablet computers and smart phones.These smart devices are deeply involved in human life,and have largely changed the way humans communicate.In this fast-paced society,people tend to use sketches,which are concise and rich in information,to communicate,which has also led to the emergence of a large number of sketch-related application areas,including sketch generation,sketch retrieval,sketch recognition and so on.At the same time,with the vigorous development of deep learning,it has achieved great success in the related fields of natural images,including image recognition,image generation,and image segmentation.However,the two-dimensional natural images obtained by the camera are usually perfect copies of the real world.Hand-drawn sketches are different.It is a product processed by the human brain and has a strong subjectivity.In the visual field,hand-drawn sketches are a special modality,which also determines that it is not feasible to directly apply the method of the natural image field to the sketch field.The method of the hand-drawn sketch field requires special design and ideas.In order to facilitate calculations,hand-drawn sketches in computers are usually stored as two-dimensional pixel pictures,however,this storage method will obtain a highly sparse matrix.The process of drawing sketches by humans is a dynamic process.Representing sketches like natural images,although it can retain a lot of information on the visual shape of sketches,it will inevitably lose a lot of dynamic information when drawing.The advantage of expressing sketches in vector form is that it can retain the timing information of strokes during painting.The works of this thesis on both sketch generation and sketch segmentation belong to the study from the perspective of vector sketches.The main works and contributions of this thesis are as follows:1.The first work of this thesis is to propose a vector sketch generation method based on the adversarial autoencoder framework.On the one hand,because of the need for humancomputer interaction,we want to improve the machine's understanding of the information contained in the sketch;on the other hand,the labor cost of obtaining the sketch is much higher than that of the natural image.If you can teach the machine to draw a sketch like humans,then these two problems can be solved well.However,most of the existing vector sketch generation methods are based on the framework of variational autoencoders.The variational autoencoder has blur problems in the generation of vector sketches and the generation of pixelated natural images.At the same time,the existing sketch generation methods only use a single representation of the sketch.Aiming at the problems existing in existing methods,such as scribbled results and single coding sketch information.This thesis proposes a vector sketch generation method based on adversarial autoencoder.This method merges the spatial information of the raster sketch into the process of generating vector sketches by means of the confrontation mechanism possessed by the adversarial autoencoder itself,so that the generated sketches have better category shape information.It uses both the timing information between strokes contained in the vector sketch and the shape information of the drawing objects contained in the raster sketch.The sketch generation and hidden space interpolation experiments were carried out on the Quick Draw data set,and the Ske-score evaluation index was used for quantitative measurement.The experimental results show that the proposed method can alleviate the sloppy effect of the generated results,and the generated sketches have better Visual aesthetics and a higher degree of class recognizability.2.The second work of this thesis is to propose a multi-category semantic segmentation method that merges category semantic information.The semantic segmentation of sketches is a more fine-grained sketch understanding task,and it is also the basis of many sketch-related tasks.Existing image-based methods regard this task as a semantic segmentation problem of natural images,and use an advanced convolutional neural network-based architecture to handle sketch segmentation problems.The disadvantage of this type of method is that the sketch is ignored by a Consists of a series of strokes,with certain timing and other information between these strokes;conversely,the sequence-based method treats this task as a sequence prediction problem,using relative coordinates and stroke status to encode the timing between sketch strokes And other information,and then use the recurrent neural network structure to predict the data point label,however,this type of method ignores the effect of the visual shape information of the sketch on the segmentation result.This thesis proposes a multi-category semantic segmentation method that merges category semantic information.In view of the above problems in existing methods,we choose to predict the label of each data point from the perspective of sequence while incorporating the visual shape of pixel sketches.information.Based on the framework of variational autoencoder,the encoder uses convolutional neural network to encode the visual shape information of the pixel sketch,and then sends the information to the decoder using the recurrent neural network structure to help predict the label of data point components.In order to reduce the influence of the parts with the same shape and the same drawing mode between different sketch categories on the segmentation accuracy,this thesis also adds a classification loss at the end of the encoder,which prompts the encoder to encode the semantics of the sketch categories to implicit space.A sketch segmentation experiment was performed on SPG data,and P-metric and C-metric evaluation indicators were used to quantify the segmentation accuracy.The experimental results show that the fusion of visual shape information and category semantic information is beneficial to the semantic segmentation of multiple types of vector sketches.Also verified the effectiveness of the proposed model.