Research And Application Of Cross-modal Retrieval Technology For Hot News

With the development of the Internet and social media,multimedia data has shown explosive growth.Especially in the field of news,people obtain a large amount of multimedia data from social networks,including texts,pictures,videos,and so on.Therefore,the demand for management and analysis of multi-modal data in news has increased.Cross-modal retrieval is one of the more common methods.Cross-modal retrieval breaks the limitations of traditional single-modal retrieval,realizes cross retrieval between multiple modalities,and facilitates the organization and management of multi-modal data.Although the existing research on cross-modal retrieval has made many breakthroughs,cross-modal retrieval will suffer from the shortcomings of slow retrieval speed,low retrieval accuracy,and high storage cost in the face of large amounts of multi-modal data.The hash method has attracted much attention due to its low storage overhead and fast query speed,and has been widely used in large-scale data retrieval.Most of the existing hash cross-modal retrieval algorithms are supervised cross-modal retrieval algorithms.However,in specific environments,cross-modal retrieval faces problems such as difficulty in tag acquisition and insufficient paired multi-modal data.In this thesis,based on the different specific scenarios faced by cross-modal retrieval,research on semi-supervised cross-modal retrieval algorithms is carried out.The main work is as follows:(1)A semi-supervised deep cross-modal hashing algorithm based on semantic alignment is proposed.This algorithm addresses the problem of missing data labels for cross-modal retrieval under large-scale data.It uses residual networks to extract the depth features of multi-modal data,and then uses the inherent internal connections in paired modalities(image-text pairs)to generate a similarity matrix.The similarity between the multi-modal features and the similarity between the hash codes are aligned to generate a consistent hash code.In this thesis conducted experiments on three data sets,WIKI,MIRFlickr,and NUS-WIDE,and compared them with cross-modal hashing methods such as CVH,IMH,and LCMH.Experimental results show that the semi-supervised deep cross-modal hashing algorithm based on semantic alignment improves the accuracy of retrieval compared with the existing cross-modal hashing methods.(2)Propose an unsupervised semi-paired deep cross-modal hashing algorithm.This algorithm proposes a solution to the problem of reduced efficiency of cross-modal hashing algorithm due to the missing of some modal data and unknown paired information between modals in the real world.The algorithm first trains two generators to fill in the incomplete pairwise information in order to expand the limited pairwise cross-modal information in the original data.Then use the enhanced paired data to construct a correlation graph,learn the hash function,and generate a unified hash code.This thesis conducts experiments on two data sets of MIRFlickr and NUS-WIDE,and compares them with the four semi-paired cross-modal hashing methods.Experimental results show that the unsupervised semi-paired deep cross-modal hashing algorithm improves the accuracy of retrieval compared with the existing half-paired cross-modal hashing method.(3)Designed and implemented a cross-modal retrieval system for hot news.This system is based on the B/S(Browser/Server)architecture of web server and browser,integrates data acquisition and processing module,cross-modal retrieval module and visual page display module,and provides users with the function of mutual retrieval of pictures and texts.It improves the accuracy of retrieval and meets the needs of users for multi-modal data retrieval.