Research On Image Retrieval Algorithm Based On Visual And Eye Movement Features

Content-Based Image Retrieval is a hot research area of the computer vision.In recent years,many researchers tried to solve the problem of the mismatch between the low level visual features and high level semantic concepts.This thesis devotes itself in mining the intention information of the users from the eye tracking data and performing innovation with regard to the traditional image retrieval system so that the "semantic gap" problem can be partly alleviated.The main work and contributions of this thesis are as follows:First,a novel content-based image retrieval framework with eye tracking data based on implicit relevance feedback mechanism is proposed.Our proposed framework consists of three components:feature extraction and selection,visual retrieval and relevance feedback.In the first stage,by using the quantum genetic algorithm and the principle component analysis algorithm,optimal image features with 70 components are extracted.In the second stage,a finer retrieving procedure based on multiclass support vector machine and fuzzy c-mean algorithm is implemented for retrieving most relevant images.Finally,a deep neural network is trained to exploit the information of the user regarding the relevance of the returned images.This information is then employed to update the retrieving point for a new round retrieval.Experiments on Corel and Caltech databases show that the performance of Content-Based Image Retrieval can be significantly improved by using our proposed algorithm.Second,an image retrieval algorithm based on user intention is proposed.In this method,the intention of the user is obtained implicitly by exploiting the information from the eye tracking data of the user when browsing the returned images.For mining the user intention from eye tracking data,a cascade convolutional neural network is constructed and trained.In this network,a 1-D network is trained and employed to extract the time related features from the eye tracking data while the 2-D VGG-D network is employed to extract the spatial features of the eye tracking data.Finally,these two types of features are concatenated together and input to another 1-D network which is then trained to extract the optimal spatiotemporal features.These features are then used to compute quantitatively the intention of the user with regard to the relevance of the observed image to the query one.In order to compute the relevance of any image in the dataset without eye tracking data,the eye tracking data are transferred from the user observed images to the ones(without eye tracking data)with the most similar visual features.In such a way,the intention based image retrieval can be implemented.Experimental results show that the image retrieval algorithm based on user intention can greatly narrow the semantic gap and improve the performance of the retrieval systems.