Research On Free-Hand Sketch Recognition And Semantic Segmentation Method Based On Deep Learning

With the popularity of touch screen devices and the emergence of drawing programs,there are more and more scenes that require sketching in the internet and daily life.Understanding these sketches is the key to achieving human-computer interaction.Unlike natural images,sketches are composed of several strokes.Therefore,the content of sketches is sparse and presents different levels of abstraction,making it increasingly difficult to understand them.Sketch recognition is the understanding of the entire sketch.By classifying the sketch,the content of the sketch is understood,while sketch semantic segmentation is the semantic classification for each pixel on the sketch.Sketch recognition and semantic segmentation are understood from coarse and fine granularity,respectively,and some semantic segmentation methods rely on recognition methods.In view of this,this dissertation aims to study the theories and methods of sketch recognition and semantic segmentation based on deep learning,and specifically carry out research work in the following four aspects.Firstly,in response to the current situation where sketch recognition methods with high recognition accuracy fail to handle static sketches,ignoring the impact of point modal features and image modal features on the expression of sketches,a bimodal sketch recognition method is studied using collaborative learning paradigm.In order to handle static sketches,the point modal branch and image modal branch are constructed based on convolutional neural networks.In response to the problem of difficulty in maintaining local structural information during the process of extracting local features from point modal branch,a rational division of the local region is carried out based on the distance information between the center points and their adjacent points,and a structured point convolution block is developed using attention mechanism to improve the ability of point modal branch to maintain structural information.In response to the lack of multi-scale information in the image modal branch,based on scale space theory,a multi-scale residual block is designed using the random activation method.Based on the multi-scale residual block,a hierarchical residual image modal branch is constructed to better understand image modal data.Based on the complementarity between point modal features and image modal features,a bimodal model is created to implement a sketch recognition method.Based on the different learning abilities of the point modal branch and image modal branch,combined with the characteristics of neural networks fitting non noisy and noisy samples during the training process,the learning method of the bimodal model is explored to resist noisy samples,thereby improving the performance of the bimodal sketch recognition method that do not rely on stroke sequence information.Secondly,in response to the current situation where existing sketch semantic segmentation methods do not consider the relationship between segments,and the stroke structure is not considered in the local feature extraction process,a sketch semantic segmentation method is studied based on segment self-attention."Stroke distance" is defined based on the stroke information and structure related to the center points and their adjacent points.Based on "Euclidean distance" and "stroke distance",rich position and stroke information is collected,a local feature aggregation module based on stroke structure is designed to better extract local features.A stroke is expressed as several segments,and the extracted local features and point position information are used to learn the embedding of segments in strokes and sketches.Based on the multi-head self-attention mechanism,a segment self-attention module is designed to learn and strengthen the relationship between segments.Based on the encoder-decoder structure,a sketch semantic segmentation model is constructed using the global context information of neural networks and thereby achieving sketch semantic segmentation.Thereafter,in response to the current situation where existing sketch semantic segmentation methods do not consider stroke continuity and result in the loss of semantic details during feature extraction and selection,a sketch semantic segmentation method is studied based on point-segment interaction.Based on the distance information in coordinate space and feature space,and deep learning theory,combined with the positional relationship between points,an enhanced local feature aggregation module is designed to ensure the rationality of local features.Based on the complementarity between point data and segment data,a dual branch encoder-decoder semantic segmentation structure is established using an encoder-decoder structure,combined with the sparsity of sketch content.In response to the issue of detail loss in the feature extraction and feature selection process,drawing on the multi-head self-attention mechanism,the correlation matrix between points and segments is calculated.Based on deep learning theory,a point-segment interaction module is constructed from the channel and spatial dimensions to complete the sketch semantic segmentation task.Finally,sketch semantic segmentation methods typically rely on stroke sequence information.In response to the current lack of stroke sequence information in most sketches and the fact that existing sketch semantic segmentation methods do not consider the overall semantics of sketches,a sketch semantic segmentation method is studied based on conditional generative adversarial and mean teacher model.Based on the continuity of strokes,a pseudo stroke searching algorithm is designed.Based on the pseudo stroke information,an enhanced local feature aggregation module is used to extract local features from sampling points in sketches,avoiding dependence on stroke sequnce information.Using the local features of sampling points as graph structure nodes,using the proximity between point positions and connectivity between points,the relationships between nodes are established,and based on these relationships,a relational graph convolutional neural network is established.Based on the generative adversarial idea,a generative model for sketch semantic segmentation is constructed using a relational graph convolutional neural network as a generator,combined with an enhanced local feature aggregation module and a multi-layer perceptron.A repetitive training method is designed to improve the semantic segmentation effect through the generated semantic segmentation maps.Based on the enhanced local feature aggregation module and the mean teacher model,a sketch semantic classification model is established.Based on the semantic segmentation maps output by the conditional generative adversarial network,the incorrect semantic segmentation results are corrected using the semantic types output by the mean teacher model,improving the overall semantic impact of the semantic segmentation method,and achieveing sketch semantic segmentation without relying on stroke sequence information.