Research On Multimodal Data Fusion Methods

In the big data era,a wide array of data have been generated in multiple modalities.Generally speaking,different modalities represent data samples from different perspectives which usually provide complementary information to each other.However,how to exploit the complementary characteristics and unlock the huge value hidden in the mass is a tough problem that big data research focuses on,meanwhile it is the major difference between big data and traditional data learning tasks.Data fusion is a powerful tool for multimodal data analysis and mining,but the characteristics of incomplete modality,real-time processing,modality imbalance and high-dimension attributes for multimodal data pose great challenges to the design of data fusion methods.Aiming at these challenges,this dissertation studies the corresponding schemes,such as incomplete multimodal analysis,incremental multimodal co-clustering,heterogeneous modality transfer learning and low-dimensional feature sharing,for multimodal data fusion.The main contributions are as follows:(1)The existing incomplete multimodal analysis methods for data fusion may not be effective enough to facilitate the semantic sharing between modalities.To tackle this problem,a new incomplete multimodal data fusion method via deep semantic mapping is proposed.By exploiting the high-level semantic abstraction characteristic of deep neural networks(DNN),a unified deep model integrating modality-specific DNN with incomplete multimodal common feature learning is designed to boost the correlating and fusion of incomplete data,thus it can reduce the semantic bias for shared features.Besides,an affinity graph based regularizer is constructed to couple the shared features in common subspace and the original features in each modality,which can preserve the local invariance for different modal space and improve the accuracy for fusion results.Experiments validate that the proposed method can learn the deep semantic mapping subspace effectively for incomplete multimodal data,and thus it can guarantee the accuracy of data fusion results.(2)The existing incremental multimodal co-clustering methods for data fusion may be easily affected by the selection of parameters.To address it,a parameter-free multimodal incremental co-clustering method for data fusion is proposed,in which a new multimodal similarity measure is defined and three clustering strategies,namely cluster creating,cluster merging and instance partitioning,are designed to incrementally integrate new arriving objects to current clustering patterns without introducing additive parameters.Thus the efficiency and robustness of clustering can be improved.Moreover,with the increasing of data samples,the effect of different modality on the clustering results may be different,so an adaptive weight scheme is designed to measure the importance of feature modalities dynamically in incremental co-clustering,which can enhance the scalability of the proposed algorithm.Experiments validate that the proposed method can guarantee the clustering accuracy for new arriving multimodal data,meanwhile it can improve the efficiency for dynamic data fusion.(3)The existing heterogeneous modality transfer learning methods for data fusion may not be effective to bridge the gaps when there is a large distribution divergence or feature bias between modalities.To address this problem,a multilayer semantic mapping based heterogeneous modality transfer model for data fusion is proposed.It integrates the deep neural networks with the semantic correlating model to design a unified multiple-layer semantic mapping model across different modalities.By the cross-modality correlation mapping for each layer,the semantic bias between heterogeneous domains can be reduced gradually.Moreover,a top layer correlation matching between modalities is exploited to fine-tune the whole network in back propagation,which can further improve the correlation across domains.A new objective function is defined to jointly optimize the proposed multilayer deep semantic mapping network,thus to achieve the high-level semantic fusion subspace,where the knowledge of the source modality can be transferred for task learning in the target modality.Experiments validate that the proposed model can effectively bridge the gaps between heterogeneous modalities through multilayer correlation mapping,therefore it can achieve more promising transfer results than the state-of-the-art methods for data f;usion.(4)The existing low-dimensional multimodal feature sharing methods for data fusion may not be effective to eliminate the influence of modality-specific information in sharing feature learning.To tackle it,an unsupervised multimodal non-negative correlated feature fusion method is proposed.It learns modality-specific(uncorrelated or negative correlated)features and captures inter-modality feature correlations in the latent common subspace simultaneously.By separating the modality-specific features,it can improve the validity of low-dimensional correlated features.A multimodal joint objective function is constructed based on the shared cross-modality features.Moreover,the invariance graph regularizer and sparseness constraint are integrated to boost the optimization processes,thus the accuracy of fusion results can be improved.Through iterative correlated and uncorrelated feature co-training and updating,the robust cross-modality shared features in low-dimensional subspace can be achieved.Experiments validate that the proposed method is effective for low-dimensional correlated features fusion and promising for high dimensional multimodal data reduction.