Deep Learning Representation Based Image Retrieval

Recent years have witnessed the prevalence of digital cameras and smartphones,as well as a consistent growth in terms of the capacity of storage devices.Along with such innovations,there comes an explosive growth of multi-media contents,especially visual contents,on the Internet and even on a single storage client.As a result,efficient and effective retrieval methods for massive visual contents become the hot spot in both research and industry communities.The early image retrieval systems usually apply text contents as the query,and the images related to web text contents which match with the query are returned as the retrieval results.The development of computer vision technology opens a new perspective for image search,i.e.,content-based image retrieval(or CBIR).This retrieval method complements the search engines with deeper understanding of user intentions,and enriches the user experiences.Meanwhile,it has been broadly applied to the Internet applications such as product search,landmark locating,and trademark duplication checking.As deep learning architectures,especially Convolutional Neural Networks(CNN)has remarkable power of image representation,plenty of works are dedicated to applying deep learning to image retrieval systems.However,such deep representation suffers from some issues regardless of prominent retrieval performance.First,unlike traditional local invariant feature based representation,the deep representation depicts images from the holistic and semantic view,which lacks local details and is sensitive to spatial geometry variances.Furthermore,local invariant feature based methods can exploit geometry verification for more accurate matching,while deep representations can hardly use this property to improve retrieval precision.Apart from those,existing methods usually test the performance on several public benchmark datasets with manual annotations and cannot assess the retrieval quality for arbitrary queries on-the-fly,which is important for search engines to revise the retrieval results as needed.In this thesis,we focus on the deep representation based image retrieval technology and investigate effective representation method,retrieval performance enhancement,and real-time retrieval quality assessment.The contributions of this thesis include:(1)We propose a compact deep image representation based on general object detection to fully exploit the semantic representing power of deep learning along with the discrimination of local salient image regions.We first apply a general object detector to detect a few regions that may contain objects,and then extract deep features in these regions.To integrate local properties in the regions,we extract local invariant features and combine them with the deep representation to produce a more robust representation.(2)We propose to promote the performance of deep representation based image retrieval with database boosting and query result re-ranking techniques.With minor computational and storage costs,the improvements are achieved in off-line indexing phase and online query phase,respectively.For off-line database boosting,we make use of neighborhood relationships between database images to update the feature rep-resentations.For online query,we design a local residual representation for top-ranked images in the initial retrieval result,and re-rank these results.Both processes aim at improving the retrieval accuracy.(3)We propose a correlation based method to automatically assess the retrieval quality online and demonstrate applications such as retrieval result selection.For each retrieval result,we compute a correlation based feature matrix with the deep repre-sentation of images,and then use a convolutional neural network to learn a regression model for retrieval quality prediction.Multiple features can be fused together for robust representation.To conclude,this thesis conducts a comprehensive investigation on deep represen-tation based image retrieval in terms of representation construction,off-line indexing,online re-ranking,retrieval quality assessment,etc.We evaluate our study with experi-ments and applications and demonstrate the reliability and practicability of the methods.