The Research Of Sketch Recognition Combining With Semantic Information

With the emergence of many smart devices such as mobile phones,tablet com-puters and smart pens,free-hand sketches are more frequently seen by the public.Sketch images are highly abstract and easy to draw,expressing rich meanings,so they are popular with the public.However,due to the different painting skills of the free-hand sketchers,the sketch is also presented in a variety of forms.For the same object,different sketches drawn by different people are also very different,which is why the existing annotated sketch dataset is very limited,especially in certain fields.In the face of such diversified sketches and such a shortage of sketch dataset,how to make the computer accurately understand free-hand sketches and how to effectively identify free-hand sketches has become an increasingly important research problem.Understanding sketch accurately and recognition sketch efficiently are crucial for human-computer interaction and various sketch applications in daily life.How-ever,due to the simple structure,single color and various forms of expression of the sketch,there is a huge semantic gap between the original sketch and its high-level semantics.In order to make up for the semantic gap between them,so that the com-puter can more accurately and efficiently understand and recognize the sketch,this paper deeply analyzes the structural characteristics of the sketch itself,and integrates semantic information into the sketch recognition.The semantic information of sketch in this paper is divided into two categories.The first one refers to the seman-tic of sketch labels.The ground truth label of each sketch or its'components are real and semantic.The second one refers to the semantic of sketch shape,different com-ponents consist of sketch,and the same component has similar shape,various shape consist of the sketch.Studied on other mainstream feature extraction methods and sketch recognition methods,we deeply combined the sketch of semantic information.This paper proposes a sketch recognition method based on semantic tree,and a nov-el feature representation of sketch based on the fusing of sparse coding and deep learning,which verified on the sketch recognition accuracy.The main work and in-novation points of this paper are as follows:1.Based on label attributes of sketch parts,this paper proposes a sketch recog-nition method based on semantic tree.This method proposes a new network model(sketch-semantic Net)aiming at the non-interpretability of the recognition process,which caused by treating the whole image as the input of framework.This method firstly adopts sketch segmentation,splits the whole sketch into multiple single com-ponents with semantic concept,then transfer the convolutional neural network trained in the natural image to retrain the sketch components,lastly,related the re-sult of the new sketch network to the semantic tree,finding the relationship with the sketches' components' concept and the sketches',which effectively compensate for the semantic gap between low-level semantics to the high-level semantics.The main innovations of this method are as follows:1)use the components' specific sematic information of the sketch to help the recognition of the whole sketch image,and in-tegrate the deep learning with the semantic tree;2)in the fusion strategy,con-text-based semantic fusion strategy is adopted to effectively alleviate the influence of semantic ambiguity on fusion.Experimental research shows that the proposed method is effective on the dataset(Sketch_Dataset).2.Based on the semantic of sketch shape,this paper proposes a new sketch feature representation method which integrating sparse coding and deep learning.In view of the poor ability of sketch feature representation in current sketch under-standing,this method proposes a new Algorithm named Sketch-Representation Al-gorithm(SRA),to solve the problem that when the scale of datasets in some specific fields is very limit,and that purely use the manual extraction representation methods or use machine learning methods would affect the performance of the sketch recog-nition.The algorithm firstly to combine the semantic components sketch and then transfer the convolutional neural network trained in the natural image to retrain sketch components and whole sketches respectively,and then reduce the dimension of both.For components' feature,we firstly use the K-means to get their center.Secondly,treat the centers as the basic dictionary of the sparse coding.Lastly,use the sketch feature trained from CNN and the centers to alternate the new parse cod-ing model.In this way,by adding the semantic information to the sketch feature representation,those representations full of interpretation.Different from the previ-ous sketch feature representation methods,this method introduces the features of sketch components trained from deep learning into sparse coding and treat it as the initial basis vector of the dictionary,introduces the semantic information into sparse coding,which improve the performance of sketch feature representation meanwhile make sparse representation more interpretable.The sketch feature representation method has achieved considerable results in sketch recognition,and the experi-mental results show that the algorithm is effective.