Research On Social-Sensed Cross-Modal Retrieval

With the development of information technology,the resent years have witnessed the explosive gro.Awth of cross-modal data,such as text,image and video,which have the same semantics.Users can produce,share and disseminate these data anytime,anywhere,forming a close,and complex social interaction between users and data.Therefore,the cross-modal data and users do not exist in isolation,they all have clear "social" trends.Analysis and mining of the correlation between different modalities for cross-modal retrieval has become an important research topic in this area.However,most of the existing methods study the cross-modal data in isolation and don't deal with the complex semantic correlation due to the"socialization",so it can't solve the heterogeneous gap effectively.In addition,the existing research on cross-modal retrieval focuses on learning the correlation matching,however whether such matching is required by users doesn't considered.That is to say,there isn't analysis of individual needs,so it can't solve the user need gap,which makes it difficult for user to get required information.Hence,it is of great significance to fully exploit the semantic correlation between cross-modal data and understand the needs of individual users to improve the performance of cross-modal retrieval.This research is based on the co-construction project of scientific research and postgraduate cultivation supported by Beijing education committee,which is analysis and mining on social-sensed cross-media data.This paper investigates the problem of heterogeneous gap and intention gap in depth using the sensed social information,aiming to improve the retrieval performance from two perspectives.The main research contents and innovations include:1.In order to handle the problem of heterogeneous gap between different modalities,this paper propose a correlation learning method by combining link and content.Most existing methods study the correlation representation through the character of co-occurrence and compllementarity,which can't capture semantic correlation thoroughly.This paper use the social characteristics and sense the social information,to enhance the semantic correlation between heterogeneous data by using social correlation.In order to handle the complex correlation,this study use heterogeneous information networks to model the data,so the different semantic social correlation can be represented by network link.In order to learn the semantic correlation,this paper incorporate link-based correlation and bottom-level content into a unified framework.At the same time,learning an effective transform mechanism,so the heterogeneous data can be mapped into a common semantic subspace.What's more,the spatial heterogeneity of data can be eliminated to achieve cross-modal retrieval.Experiments on expanded NUS-WIDE dataset show that the proposed method can effectively improve the performance.2.In order to deal with the problem of need gap between cross-modal data and the information needs of users,this paper propose a model of user interest based on social multi-modal topic model.In cross-modal retrieval,users often can't express their information needs in other modalities clearly,lead to the ambiguity of query.Moreover,the traditional methods of modeling single-modal data such as user metadata,query or click log are unsuitable for cross-modal retrieval.Therefore,this paper sense the rich multi-modal data in social media to mine user interest,and retrieval needs can be expressed by long-term interest to get personalized results.In order to handle the challenges of multi-modal and noise,this study proposes a social multi-modal topic model.The model not only modeling multi-modal data simultaneously,which can be applied to cross-modal retrieval flexibility;but also take social influence of friends' interests into account,which can improve the robustness and accuracy of the interest expression.In addition,the experimental results verify the effectiveness of this method.