User Intention Oriented Social Image Retrieval

With the development of Internet,image retrieval plays an increasingly important role in information retrieval in our daily life.Traditional image retrieval methods usually focus on semantic gap(i.e.,the gap between low-level features and high-level semantics)but ignore the intention gap(i.e.,the gap between search intention in users' mind and the query input in search engine)due to the lack of personal identification information.In recent years,users share and browse images in social media platforms,which provides us an effective way to obtain personal preference information.In this thesis,we combine the social information including user profile,user relation and user behavior with visual contents in image representation level,distance metric level and image reranking level for image retrieval.Thus,we can better understand user intention through user preference and provide users a more satisfactory search result.The contributions of this paper can be summarized as follows.1.Proposing a socially embedded image representation learning approach.We combine the social behavioral information with visual content information in image representation learning stage using multi-task deep learning model.In addition,we consider the sparseness and unreliability problems in social media data in our method.The learnt image representation can capture not only semantic meanings but also user preference information.Thus,it can better understand user intention and be applied in image search and recommendation scenarios.2.Proposing a multi-relational knowledge embedded image representation learning approach.This approach embeds multi-relational knowledge graph into image representation learning problem and jointly optimizes these two tasks based on a Relational Regularized Regression CNN.The learnt representation can involve not only the tag information in image-tag data,but also the tag relations in knowledge graph.Such representation can solve the preciseness and completeness problem in image representation learning.3.Proposing a socially embedded image distance learning method.In user-centric image applications(such as image retrieval and recommendation),traditional visual content based distance metrics cannot effectively capture user intentions due to the intention gap.In this thesis,we embed the similarity of social behavioral information into visual content space and make visual similarity consistent to social similarity.Therefore,we can predict social behavioral similarity based on their visual contents.In addition,we prove that compared to traditional image distance metrics,our methods can better understand user intention in search and recommendation.4.Proposing a social-visual personal image reranking approach.For a given user,we use his/her social behavioral information in social platforms to provide personalized search results in image search engine.Facing the challenges of sparseness and unreliability of social data,as well as complex social factors,we conduct random walk method based on the hybrid graph of social information and visual contents.In addition,the effectiveness and importance of social information in personalized image search are demonstrated.