| Human cognition is supported by the combination of multi-modal information from the visual,auditory and tactile,because it is difficult to reflect the complex scenes that exist in the real world by the single-modal information solely.With the popularity of mobile devices,people can upload various kinds of information on the Internet quickly and easily.This information involves multi-modal data such as text,image and audio,etc.In the case of massive multi-modal data,there is an urgent need to find connections between data from different modality,thus giving the machine the ability to combine them to accomplish a specific task.The cross-modal feature correlation technology aims at establishing the correlation between data from different modality at the semantic level by the data itself,which can implement applications such as cross-modal retrieval for multi-modal data more accurately.Nowadays,the wave of deep learning promotes the rapid development of deep models.A large number of deep neural networks have emerged for this reason,and the Generative Adversarial Nets is one of the representative models.Under the guidance of adversarial thought,the generative adversarial nets has an incomparable advantage in learning the data features compared to the general deep models.For the cross-modal feature correlation,it requires that the correlated features are discriminative to semantics and consistent with modalities.Essentially,cross-modal feature correlation is a process of feature learning.So the adversarial thought consists in the generative adversarial nets can provide a powerful tool for cross-modal feature correlation.On this basis,this thesis improves a cross-modal feature correlation algorithm based on the generative adversarial mechanism,and builds a cross-modal retrieval system.Finally,it combines multi-modal datasets to verify its performance on cross-modal retrieval task.The experimental results show that our system performs better than other traditional models.Specifically,the main research results of this thesis are presented as follows:1.An improved cross-modal feature correlation algorithm based on generative adversarial mechanism is proposed(cross-modal adversarial correlation learning,CMACL).By using the powerful ability of feature learning provided by generative adversarial mechanism to realize the cross-modal feature correlation,conducting an improved classifier to perform a precise classification at semantic level for small sample data,the correlated features are discriminative to semantics and consistent with modalities.2.A cross-modal retrieval system based on the CMACL algorithm is constructed,and the cross-modal retrieval tasks are accomplished with the combination of multi-modal datasets.The performance on cross-modal retrieval tasks is used as a reference to verify the superiority of our improved cross-modal feature correlation algorithm.3.A small multi-modal dataset containing three types of data such as image,text,and audio is constructed.To verify the effectiveness of the improved algorithms,we also apply this dataset to the established cross-modal retrieval system. |
