Research And Application Of Cross-modal Retrieval Method Based On Feature Fusion

With the arrival of the Internet epoch,a huge amount of digital information is requested and transmitted every day.These digital information has different modal types,such as text,image,video,audio and so on.People pay more and more attention to the retrieval operation of this kind of multimodal data.In addition,due to the development of digital museum and data digitization,a large number of digital information such as images and texts have been generated with Chinese painting and calligraphy as the data source.How to better organize and use this kind of data and more effectively complete the task of cross modal retrieval has become an important branch in the field of multimodal data research.This paper takes two different modal data of text and image as the starting point,aims to fully explore the relevance between the underlying features and the high-level semantics of the two,and uses a unique feature fusion method to complete the task of cross modal retrieval.The main work includes the following:(1)In view of the particularity of Chinese painting and calligraphy data and the limitation of current mainstream methods,a cross modal retrieval method based on feature fusion is proposed.This method makes full use of the advantages of the pre training language model and convolutional neural network in the acquisition of text and image features,and proposes a feature fusion method,which combines the two organically,to complete the cross modal retrieval task of text search image and combination search of text and image.Experiments are carried out on the public dataset fashion200k and MIT states.By comparing with other fusion methods of different modal features,the effectiveness of the proposed method in improving the performance of cross modal retrieval is verified.At the same time,aiming at the data of Chinese painting and calligraphy,a set of labeled painting and calligraphy image data is constructed.The proposed method is tested on the data set to verify the effectiveness of the method in the task of cross-modal retrieval of painting and calligraphy images.(3)An interactive Java Web application is implemented.Based on the relevant data of Chinese painting and calligraphy,the system can display and retrieve the data of painting and calligraphy images,seals and related person.