Research On Sketch Recognition Using Convolutional Neural Networks

With the popularity of the Internet and the rapid development of multimedia technology, the number of images in the Internet has increased dramatically. The emergence of images with various storage methods and huge storage capacity has made a staggering database both of amount of information and quantity, which greatly challenges image processing, recognition and retrieval.In recent years, with the popularity of smart mobile phone, notebook computer and other related equipment, the rapid development of touch screen technology, the technology of sketch-based retrieval has gradually attracted the domestic and foreign scholars. However, due to variability in the line of hand-drawn sketches and paintings of different people involving uncertain and subjective factors, current hand-drawn sketches generally have problems of low recognition and poor universality.In recent years, deep learning, as one of the most remarkable research techniques in the field of artificial intelligence, has succeed in speech recognition, gesture recognition and image recognition. However, the current classical learning model of deep learning is mainly designed for colorized multi-textured natural images while hand-drawn sketches are lack in information of color, texture etc. As a result, current models like ImageNet are not able to achieve the desired effect while recognizing hand-drawn sketches.Compared to image representation based on low-level local descriptors, deep neural activations of Convolutional Neural Networks (CNNs) are richer in mid-level representation, but poorer in geometric invariance properties. Fisher Vector has a great power in geometric invariance properties.Recently, Schneider and Tuytelaars demonstrated that Fisher Vectors, an advanced feature representation scheme successfully applied to image recognition, can be adapted to sketch recognition and achieve a good accuracy, especially when it is combined with BOW. But it considers neither the features of hand-drawn sketches nor the order of sketched strokes.Based on intensive study, the main innovation points of this essay are as follows:(1)Most of sketch recognition methods still heavily rely on the hand-craft feature extraction techniques which are time-consuming and pretentious, and the ability to extract better features depends on experience and luck. Meanwhile, the model trained on ImageNet fails on recognizing the sketches, for the small-sized filter in the first layer can be recognized as eye or button in a photo based on texture, but sketches lack texture information. In order to solve this problem, this paper proposed a method of sketch recognition based on deep learning, called Deep-Sketch. The classical deep learning models were mainly designed for natural color image recognition which failed on the sketch recognition. Deep-Sketch aimed to obtain more spatial structure information by using large-size convolution kernel instead of the small-size convolution kernel in the first convolution layer. In addition, a shallow model was trained to get parameters which were used to initialize the corresponding layer parameters of the Deep-Sketch to reduce the model training time. The model was deepened with the convolution layers which kept the feature size to reduce error rates. The results showed that the Deep-Sketch was superior to other state-of-the-art sketch recognition methods and achieved 69.2% accuracy on 250 kind of hand-drawn sketches.(2) Compared to conventional image recognition methods based on low-level local descriptors, convolutional neural networks are richer in mid-level representation, but poorer in geometric invariance properties. In this paper, we present a framework for better image representation by combining the two approaches. We use fisher vector, which is strong discriminative, to take the activations from a pre-trained CNNs. It can rich the feature representations and reduce the complexity of the training of classifier. We add stroke order to distinguish classes which are similar in parts but differ in ordering for stroke order can express how people understand objects. We utilize data augment to prevent over-fitting and enhance the geometric invariance. Experiments demonstrate that our method can be used as an outstanding representation for better performances in sketch recognition tasks.