Research On The Audience Oriented Affective Content Representation And Recognition Methods In Film

With the proliferation of digital audiovisual, the challenge of extracting meaningful content from such data sets has lead to research and development in the area of content based video retrieval (CBVR). An important and often overlooked aspect of human interpretation of video data is the affective dimension. To address this problem, video affective computing is proposed, which is one of the latest research areas and can utilize both CBVR and affective computing theories to understand video affective content. However, due to the inscrutable nature of human emotions and seemingly broad "affective gap" from low-level features, there is still lacking a unified theoretical framework for video affective content understanding. Taking the film as study object, a solution for audience oriented affective content representation and recognition is presented.To represent film affective content effectively and describe the personalization of audience faithfully, an audience oriented film emotion space is proposed. It can unify the discrete and dimensional emotion model by introducing the typical fuzzy emotion subspace. Fuzzy C-mean clustering algorithm is adopted to divide the emotion space. Gaussian mixture model is used to determine membership functions of typical affective subspaces. At every step of modeling the space, the inputs rely completely on the affective experience recorded by the audiences. The advantages of the audience oriented film emotion space are the personalization, the ability to define typical affective states areas in the emotion space, and the convenience to explicitly express the intensity of each affective state. The experimental results validate the model and show it can be used as an audience oriented emotion space for film affective content representation.To bridge the "affective gap" between low-level features and film affective content, a set of film affective features are designed, extracted and selected. These film affective features are designed according to the theories of emotional psychology and filmmaking. Whitney feature selection algorithm is implemented and two sets of film affective feature vectors are formulated. One is for describing the affective valence and the other is for describing the affective arousal. The comparative experiments show that the proposed film affective features outperform the existing studies in classifying the positive and negative of the affective valence and arousal.To recognize the film affective content, the affective highlight in the film should be detected in the first place. A multilevel film summary is proposed based on the arousal curve and film affective tree (FAT). The arousal curve indicates how the intensity of the emotional load changes along a film, and depicts the expected changes in audience's arousal intensities while watching that film. Film affective units (AU) in different granularities are firstly located by arousal curve, and then the selected affective content units are used to construct the FAT. The AU at each level of the FAT can be organized as a film summary. Two methods are proposed to recognize the affective content of AU in the summary, which are the genetic algorithm combined hidden marcov model (GA-HMM) based affective content recognition and the emotion space based affective content recognition. The first method can be used to recognize the basic emotional events of audience. The experimental results show that GA-HMM can achieve higher recognition rate with less computation compared with classic HMM. The second method adopts multi-layer perceptron and multiple linear regression to compute the emotion coordinates of the AUs in film summary. Based on the affective membership functions and emotion coordinates, the maximum membership principle and the threshold principle are introduced to represent and recognize the emotional preferences of the audiences. Experimental results demonstrate that this method can effectively represent and recognize the personalized film affective content.There are many research issues exist in the audience oriented affective content representation and recognition. The proposed film emotional space depends entirely on the affective evaluation of the audience, which is a tedious and heavy burden to the user. How to make good use of existing data to service other users is an important research issue in the future. Because of the relation between emotion and audio-visual is still unclear, the selected film affective feature vectors are not perfect for classifying the positive and negative of the emotional valence. Further investigation on the domain knowledge should be implemented to design the more reasonable film affective features. Furthermore, the emotional information of the audiences is not comprehensive enough. To describe the information of the audience's emotional personality more accurately, the coverage of the audience should be further expanded.