Sketch Recognition Using Deep Learning

With the rapid development of Internet information technology and multimedia technology,the number of images on the Internet has been growing at an amazing rate,including both physical images that are common in life,and many hand-drawn sketches.In addition,with the development and upgrading of smart phones and laptops,the resolution of images becomes higher and higher,so the space needed to store image content becomes larger and larger.Faced with such a large number of image data,it is almost impossible to manually annotate the semantic content of images.Therefore,how to realize the high efficiency labeling and recognition of massive images is an important project in the field of image.In recent years,sketch recognition task is gradually getting more related research scholars' interest and attention,and it also gradually occupies an important position in the field of image research.On the one hand,the rapid popularization of portable touch devices makes it easier and more convenient to obtain hand-drawn sketches;On the other hand,it is easier to understand the information conveyed by hand sketching.Sometimes only using simple strokes can express semantic information that is difficult to describe in text.However,sketch recognition task is still a work full of difficulties and challenges,the reasons can be attributed as follows:(1)hand-drawn sketches have two properties that are not conducive to recognition,one is highly,and the other is symbolization;(2)because of the differences in the level and ability of each individual's painting,the objects of the same category may be far apart in appearance and abstraction;(3)there is no visual clue in hand-drawn sketches,such as color and texture information.The early sketch recognition basically follows the traditional image recognition method,which is to extract the manual features from the hand-drawn sketches,and then put the features into a classifier.Some common manual features include histogram of oriented gradient(HOG)features,scale invariant feature transform(SIFT),and shape context features.However,these manual features are mainly designed for natural images and are not entirely suitable for hand-drawn sketches which are abstract and sparse.In recent years,deep learning is undergoing rapid changes,and various classical deep learning models are born.In the research of image understanding and speech recognition,they are very eye-catching.But these classical deep learning models are designed primarily for natural images whose color information and texture information are very rich.However,there is a lack of visual clue such as color and texture in hand-drawn sketches,so they are not suitable for sketch recognition.At present,some researchers have proposed deep learning models applicable to sketch recognition.These models are designed according to the unique structural features and properties of hand-drawn sketches,which can be used to obtain good sketch recognition results.But these models ignore the sequence of strokes in hand-drawn sketches.Deep learning methods generally rely on a large number of training data to suppress the influence of overfitting to obtain good recognition performance.However,the largest public hand-drawn sketch dataset currently has only 20,000 hand-drawn sketches,and it can have a great negative impact on the training of the model due to the lack of training data.But this problem can be solved by referring to the idea of transfer learning.Transfer learning allows different distribution between training data and test data,it can efficiently find some public structure and characteristics between the source domain and the target domain.Thus,a lot of knowledge can be transferred from existing data to quickly establish knowledge and capabilities in another new field.At present,the proposal of transfer learning is of epoch-making significance.It is a very important and useful learning method,which has been widely studied and applied.In this paper,the application of deep learning method in sketch recognition is studied deeply,and two deep learning models are proposed based on the theoretical basis of a large number of scholars.The main work and innovation points of this paper are listed as follows:1.The existing sketch recognition methods ignore the stroke order information in extracting the feature of the hand-drawn sketch.This chapter takes the advantage of the stroke order information of the hand-drawn sketch and proposes a sketch recognition method based on deep convolutional-recurrent neural network,which combines the deep convolutional neural network and recurrent neural network.Firstly,the proposed method extracts the strokes of the hand-drawn sketch in sequence and obtains an ordered sequence of subsketches with increasing number of strokes.Secondly,a deep convolutional neural network is adapted to extract the feature of each subsketch in the ordered subsketch sequence and an ordered feature sequence is generated.Finally,the ordered feature sequence is input into a modified recurrent neural network,which constructs the temporal relations among the different subsketches of the same sketch to improve the accuracy of the sketch recognition.2.In view of the lack of training data in the current hand-drawn sketch datasets,this chapter introduces the deep transfer learning and the multi-granular sketch information into the sketch recognition.Transfer learning can not only break the limitation of the lack of training data,but also reduce the time complexity of model training greatly.Different granular sketches contain different levels of semantic information and content,and play a different role in the application of sketch recognition.This chapter also puts forward a new fine-tuning strategy,which includes two rounds of fine-tuning.Different amount of data is used to ensure the validity of the model.At the same time,the parameters of different number of network layers are adjusted according to the granular level of the sketch.The experimental results on TU-Berlin sketch dataset show that the model proposed in this chapter can effectively improve the recognition accuracy of hand-drawn sketches.