Multimodal Data Analysis And Applications Based On Multimodal Resonance And Co-Occurrence

Multimedia documents are the combination of model datum, such as audio, image, text, video (continuous vision frames) and so on. Different modal datum have different points of expression emphasis, audio modal datum pay more attention to hearing information and image modal datum pay more attention to vision information, while text model datum emphasis more on literal information, but all express rich semantic information.For better analysis and understanding of multimedia document’s semantic information, we need to fully mine the complementarities and correlation between multimodal data. So it’s urgent and meaningful to do research on multimodal data analysis.Because of the different emphasis of different models, it brings the difficulties of multimodal data analysis. The difficulties contain two aspects, one is how to define the correlation, that’s to say which objects have correlations between each other, the other is how to measure the degree of correlation. Because of the heterogeneity of modals, represented by feature extraction methods, dimension and property, there are few methods to compute the correlation between different modals directly. The heterogeneity and incomparability of modal data cause the so called "semantic gap" further and increase the difficulty of multimodal data analysis.This paper focuses on the research of multimodal data analysis, which contains movie visual and auditory attention correlation analysis and multimodal web page correlation. First, we build the movie visual attention curve and movie auditory attention curve for the movie separately, based on these two curves, we find the most correlated region in the movie. Second, web pages contain typical modals, text, image, audio, video and so on. We explore the co-occurrence relationships of multimodal data from web pages, thenautomatically recommend web pages based on web page multimodal correlation and propagation. Finally, the experiment results demonstrate our methods are useful and robust for multimodal applications.