Study On Some Key Technologies Of Integrated Multimedia Information Retrieval

In this paper, the research on multimedia retrieval (particularly the image/video retrieval), including fundamental approaches, related techniques, and typical systems, is summarized. The research issues posed by information retrieval in digital libraries or similar large-scale information repositories are also discussed. The work presented in this paper extends the conventional multimedia retrieval research by proposing three key techniques: a relevance feedback learning method reflecting spatial and temporal characteristics, a cross-reference index aided multi-modal data retrieval mechanism, and multi-channel multimedia documents retrieval and management.The spatial and temporal characteristics of relevance feedback information in content-based image retrieval are analyzed, based on which a novel relevance feedback learning method, namely, TSC-SVM is proposed. The method treats the positive and negative examples returned at different rounds with different weights in the learning process according to the spatial and temporal characteristics of the relevance feedback information. With these features, the proposed TSC-SVM method improves the performance of the learning significantly (especially for a complicated image database) as compared with the conventional SVM method. Moreover, the proposed method can capture the query concept more quickly and accurately as compared to the conventional SVM method, and is especially effective when a query concept shift occurs during the query.A novel index method, cross-reference index, is proposed to describe the semantics of multimedia objects. The method describes a multimedia object through semantically relevant multimedia objects. A cooperative mechanism is proposed through which the cross-relevance indices and low-level features are integrated to facilitate the multi-modal data retrieval and relevance feedback. In addition, a unique relevance feedback technique is developed to update cross-reference indices by learning from users' behaviors and to enhance both the long-term and short-term retrieval performance in a progressive manner.A multi-channel (corresponding to multiple modals, e.g. text, image, video, etc.) retrieval system of multimedia documents is proposed and implemented. A novel framework model is proposed to describe the contents of multimedia documents. In the framework, both the content-based features in each channel extracted from a multimedia document and the links between multimedia objects are recorded. Moreover, a graph-based cross-reference knowledge base is proposed to store the semantic relation between multimedia objects. An algorithm for semantic context analysis is proposed to calculate the semantic similarity between each object and a query. This algorithm not only endues content-based multimedia information retrieval method with semantic information, but also provides a flexible query mode, which enables a user to interact with the system through channel exchange and relevance feedback. It is demonstrated in the experiments that the retrieval performance (e.g., the content coverage rate) for multimedia objects in the proposed multi-channel retrieval system is much better than that of a system with conventional content-based retrieval methods.