Perception, Thought, Action: Research On Hierarchical Video Content Analysis Technology

Content-based video analysis is an essential approach to overcome information overload. Recently, more and more researchers are engaged in this field. My research focuses on effective techniques for video analysis to accommodate users'needs for convenient video retrieval and quick decision making. This paper proposes a"Perception, Thought, Action"based hierarchical framework for video content analysis based on the theory of cognitive psychology. Moreover, it explores deeply the following three topics: human attention model, semantic event detection and personalized video highlights recommendation. The innovations of the paper lie in the following three aspects:1. The hierarchical framework with great generalization abilityDifferent from the video genre-centered frameworks, this paper proposes a"Perception, Thought, Action"based hierarchical framework for video content analysis based on cognitive psychology. Essentially, this framework can not only integrate most of video content analysis techniques depending on different layers of human cognition but also overcome various difficulties, such as multiple video types and sophisticated rules of video making.2. Main problems and the corresponding solutions in the process of human cognitionFirstly, the main problem of Perception is attention. This paper proposes the Weber-Fechner Law-based human attention model to extract video highlights. Secondly, the main problems of Thought are concepts and logical reasoning. Depending on the two basic methodologies of human inference (deductive reasoning and inductive reasoning) this paper proposes three important methods for semantic event detection: the rule-based method, the statistical model-based method, and the decision fusion-based method, to realize the automatic annotation of semantic events. Thirdly, the main task of Action is motivation. To facilitate information access, this paper presents the personalized recommendation approach for video highlights by integrating both video content and user preference.3. Application-driven video browsing systemsTo analyze the importance of each hierarchy, we developed three systems: the hierarchical video browsing system with saliency annotation, the hierarchical video browsing system with semantic annotation and the personalized recommendation system for video highlights with video structuring information and the main techniques in different layers of the proposed framework.In this paper, we utilized objective and subjective evaluation methods for the entire framework and its core techniques. The video text extraction method and semantic event detection methods are evaluated with the criteria of Precision and Recall. The experiment results show the accuracy and robustness of the proposed methods. The performance of the proposed framework and three video preview systems are evaluated with users'feedback. By comparing this to the traditional linear video browsing system, we came to the following conclusion: with the proposed hierarchical framework, the computer can automatically analyze the saliency and semantics of video content. Moreover, it can recommend video highlights by integrating users'preference. Consequently, users can intuitively have an overview of the whole video and get candidates for preview with personalized recommendation lists. Therefore, the personalized recommendation system for video highlights can provide friendly service for video navigation and accommodate users'needs for convenient video browsing in an individualized manner.