Image Semantics Extraction And Retrieval Based On Deep Learning

With the prevalence of the Internet and the smart phones, the users upload a large number of images on the web. It’s difficult for users to find what they really need from the sea of images. It’s also not easy for the Internet company effectively integrate its massive image data resources. However, for the traditional content-based image retrieval, images are indexed by their low-level visual features, in which one key problem is the semantic gap between low-level features and high-level semantic concepts. Therefore, semantic-based image retrieval, which has been proposed to solve the semantic gap, becomes the key technical problem in the field of image retrieval.Deep learning is inspired by the primate visual mechanism. It’s a process of the iteration step by step. Self learning feature is the biggest advantage of deep learning.Deep learning is data driven. The feature is more and more complex and abstract layer by layer, from the edge feature to structural feature or high-level feature. To solve the semantic gap, this thesis designs a deep learning network which is used to project the low-level visual features to high-level features. It also studies how to measure the similarity of two high-level features. It’s a preparatory work which helps us to implement the semantic-based image retrieval system and sufficiently use the large number of images. The main contributions of the thesis are as follows:(1) To solve the semantic gap problem in content-based image retrieval, we present a deep learning network to extract the high-level features. Deep learning’s hierarchical process(low –level edge features, mid-level structural feature, high-level object feature and so on) is abstracted as a semantic model.(2) Deep learning means big and deep neural network. It needs a large number of data and label. In the real situation, the label resource is rare. This large network cannot be convergence with rare label by supervised training. So we present an unsupervised training algorithm to train the network. First, we use sparse denoise autoencoder to unsupervised learning some convolution kernels(the weights and bias between input and hidden). Second, we copy these learnt kernels to the convolution network based on the theory of transfer learning. However, we found that the number of features’ contribution to performance is very small when it is more than the threshold. Meanwhile, the size of pooling layer has an important influence on performance. Finally, we present an unsupervised method to improve the classification result by going deep and combining multistage classifiers in a committee with a small amount of features at each layer. The network is trained layer-wise via denoise autoencoder(d A) with L-BFGS to optimize convolutional kernels and no backpropagation is used. We can take the network as an image semantic feature extractor. According to the image similarity measure method,we can get the similarity between images and achieve image retrieval.(3) Based on deep learning, we present a new image multi annotation method. First,we train a deep learning network as a semantic feature extractor. Second, we use image retrieval technology to get a set of most similarity images. At last, we present a merge method to annotate the query image. In addition, we can improve the abstract ability of the network by increasing the depth of it. However, it may have gradient diffusion issue.So we try some CNN networks and finally introduce the multistage cascade into the neural network to improve the robustness of the system and the accuracy in semantic annotation.(4) At last, we design and develop an image retrieval system based on deep learning and semantic contents.