| This dissertation presents a solution to problems arising from the demand for fast information access and for sharing in real-time multimedia transmission over the Internet. Our solution exploits software agents that are placed throughout the network environment. These hierarchical video analysis agents process multimedia streams in real time, and automatically decompose and understand the multimedia content so as to facilitate information access and sharing.; Multimedia content contains both the perceptual content such as color, motion, or acoustic features and the conceptual content, which is specified based on concepts or semantics that can be expressed by text descriptions. Both types of contents are embedded simultaneously in multimedia streams, and usually are complementary to each other. This dissertation adaptively analyzes both kinds of video contents by combining mixed media cues from audio, video and text.; First, a high-performance module for on-line video segmentation based on scene-change detection is developed. It serves as the first step of any video stream construction and analysis. To meet the high computational demand, our proposed video scene change detection algorithms are very efficient while maintaining high accuracy and recall rates for fast on-line video analysis.; Second, the perceptual features of audio and video data are analyzed in a bottom-up manner and integrated so as to discriminate among the different events in any video stream effectively. An efficient decision-tree learning algorithm is used to induce a set of if-then rules which link perceptual features with the video conceptual semantic contents. These rules not only serve as a video classifier, but also guide on-line real-time video/audio feature extraction and data redistribution. A novel knowledge-based system, where knowledge is stored as learned rules, is proposed to serve as a video semantic inference/classification engine.; Third, we propose a hierarchical video categorization scheme based on machine learning of the text information contained in a video—a scheme which provides a good complement to the video/audio classification subsystem. The learned text features for each video category are also stored in the knowledge base. To fuse the text classifier and the audio/video classifier, a media cue optimizer that is trained by using cue probability distributions based on the concept hierarchy is adopted to guide real-time media query and analysis. The integration of the hierarchical video analysis, clustering and classification allows large amounts of multimedia data to be organized and presented to users in an individualized and comprehensible way, and thus facilitates easy access to online video data. |
