Research On Expression And Speech Bimodal Emotion Recognition Of Children

In recent years, emotion recognition research based on biological signals (such as facial expression images, voice signals, EEG, etc.) has received wide attention of domestic and foreign researchers, and has become a research hotspots in the field of affective computing, pattern recognition, computer vision, etc. Expression, voice and other biological signals are important medium for human's emotion expression. Researchers analyse these biological signals by designing some features and algorithms to make the computer has the ability to recognize human emotional states. Although single-mode emotion recognition research has made great progress, the current emotion recognition studies are most performed for single biological signal (modal), such as expression, voice, or EEG, etc. Compared to single-mode, two or more modes have more emotional information, therefore, the in-depth excavation and integration of various modes of biological signals, is an effective way to further rich emotion recognition research and make more effective emotion recognition method. This paper mainly studies the issue of bimodal emotion recognition based on voice and video signals, and has done some work in the bimodal data collection and collation, bimodal feature extraction and classification, etc.the specific contents are as follows:(1) Create a children bimodal emotion database. The existing bimodal emotion databases all take adults as collecting objects. In academic circles, there isn't the bimodal emotion database targeting children. Using the platform of Child Development and Learning Science, the ministry of education key laboratory, the laboratory psychology teacher design professional emotion induced experimental paradigm for children, and collect 26 children's bimodal emotion data of voice and expression. The data is marked emotion labels by voting.(2) Summarize feature extraction techniques in single-mode emotion recognition based on expression or speech, detail the frequently-used features of expression and speech emotion recognition. At the same time, do some simple single-mode emotion recognition experiments on multiple facial expression and speech databases. In addition,to make the children bimodal emotion database (1) be more easy to study,combined with the above expression and speech features, the benchmark recognition performance of the database is given.(3) Propose a algorithm based on Bi-Sparse Linear Discriminant Analysis (BSLDA) to fuse voice and expression features. BSLDA algorithm first uses the emotional labels information of voice and video data to construct a semantic feature space, and then the voice and video features are simultaneously projected onto the semantic feature space. In the semantic feature space, projected voice and video feature sets have as far as possible similar distribution, which can be effectively further fused. Fused features usually have strong emotional discriminating ability. In addition, in the process of feature projection, BSLDA also can respectively select the important features of each mode according to their features' contribution for the emotion classification.(4) Present a bimodal emotion recognition methods via the fusion of global and local information. Existing bimodal emotion recognition research all use the global features, fewer consider the timing and local information of all modal signals. According to the characteristic that voice and video are both timing signals, the method takes both models' local information into account, proposes a multi-scale feature extraction methods to extract both models' local and global features together for bimodal emotion recognition. To further deal with the global and local features and achieve bimodal emotion recognition, an algorithm based on joint reduced rank regression model (JSRRR) is presented. According to global and local features' contribution for emotion recognition, JSRRR can learn the weight to measure the value of their contribution. Meanwhile, JSRRR can also do a second study on the weighted global and local features, select features with ability of distinguishing different emotional states of them.