| With the explosive growth of the number of videos on the Internet,users put forward higher requirements for the efficiency and accuracy of cross-modal retrieval,which promotes the development of cross-modal retrieval technology.Video contains a variety of modal information not only enrich the semantic features of video,but also bring some difficulties to extract the effective video features.How to make full use of multi-modal features in video is the focus of cross-modal retrieval research.Among them,the core of cross-modal retrieval model design is to establish intra-modal and inter-modal relationships by extracting multi-modal features in video,reducing semantic differences among multi-modal features in video,and promoting alignment among different modal features.With the application of sparse sampling technology in the process of video feature extraction,it is necessary to establish high-quality temporal features between video frame features,mine important object features contained in video frames,and improve the consistent representation between video frames.To solve the above problems,this work puts forward three methods:(1)A video-text cross-modal retrieval method based on coarse-fine-grained two-layer attention is proposed.In this method,coarse-grained attention and fine-grained attention are connected in parallel,and multi-modal features in video are extracted and processed to establish the relationship between coarse-grained and fine-grained video features respectively.Finally,a feature fusion module is used to fuse coarse-grained and fine-grained aggregate features of videos to generate semantically rich video features.(2)A method of video frame category aggregation and consistent representation learning is proposed.This method uses video frame category aggregation network to extract important object features contained in video sampled frames.An internal frame aggregation loss module is used to improve the consistent representation of sampled frame features.The transfer learning method is used to extract video features from the pre-trained model,which enhances the semantic perception ability of the model,and reduces the training time of the model.(3)This thesis proposes a method based on semantic center alignment for video-text cross-modal retrieval.This method uses the semantic center alignment module to promote the alignment of the semantic features of the same object in different modalities and narrow the semantic differences between the features of the same object in different modalities.Secondly,the BERT model with residual structure is transferred to process the video features,which not only protects the spatial semantic information,but also establishes the temporal information between the video frame features. |
