Research On The Cognitive-affective State Recognition Based On Facial Expression Spatio-temporal Features

Emotion recognition in recent years has become a hot spot in the field of patternrecognition. It is an important research issue to achieve natural human-computerinteraction. Psychology holds the view that in learning process human will experiencesome cognitive-affective states which will influence learning effectiveness. However,present computer tutoring systems lack the emotional exchange with learners, whichresults in poorer teaching effectiveness than traditional face to face teaching.Therefore, researches on how to recognize learners’ cognitive-affective states are thekeys to achieve intelligent tutoring systems with emotional abilities. Facialexpressions are the most direct reflection of human emotions. Most of currentresearches on emotion recognition extract the static spatial features of facialexpressions from images. However, facial expressions are dynamic, and in the modernhuman-computer interaction computers often obtain video data which containtemporal information. It is very difficult to use static features to obtain accuraterecognition results. Therefore, it is needed to extract facial expression spatio-temporalfeatures from video and analyze the dynamic nature of emotions to improve therecognition effectiveness. However, the high dimensionality of video and real-timerequirements of recognition put forward a great challenge to the design of emotionfeature extraction and recognition algorithms. This dissertation studies the methods ofextracting effective facial expression spatio-temporal features from video and efficientand accurate recognition for cognitive-affective states, and presents several novelalgorithms. In this dissertation, the main research contents and innovative workinclude the following several aspects:1. According to the problem of cognitive-affective states recognition based onfacial expression spatio-temporal features, the dissertation provides a comprehensiveoverview about overseas and domestic research status, studies and discuses severalkey problems, including the categorical description and the dimensional description ofhuman affects, the difference between cognitive-affective states and basic emotions,the difference between natural emotions and posed emotions, the affectiveexperimental data sets, video preprocessing algorithms (including face detection,affine transformation and histogram equalization), extraction algorithms of facialexpression spatio-temporal features (including geometric features, appearance features and mixture features), classification algorithms (including support vectormachine(SVM), k-nearest neighbor and Adaboost) and the real-time requirements ofemotion recognition.2. The dissertation studies active appearance model (AAM) algorithms, includingthe algorithms for modeling the shape and the texture and the algorithm for fittingfeatures points. On the basis above, the dissertation proposes the facial normalizeddifference deformation (FNDD) spatio-temporal features, and explains the design ideaand extraction algorithm of the FNDD. The FNDD is based on the facial shape that islocated by the person specific AAM model. The extraction algorithm first removes therigid geometric variation of the facial shape, then normalizes it on the basis of genericAAM model, last does a difference between the normalized shape and the referencefacial shape, and finally obtains the facial spatio-temporal geometric features. TheFNDD features are tested using several classification algorithms. The experimentalresults indicate that the FNDD features are effective for cognitive-affective statesrecognition.3. The dissertation studies the architecture and computing model of the moderngraphic processing unit (GPU). On the basis above, according to the problem that thehigh computational complexity of AAM fitting algorithm limits its application in realtime emotion recognition systems, the dissertation proposes a GPU parallel AAMfitting algorithm. Through the analysis of the "hot spot" of AAM fitting algorithm andthe characteristic of the GPU highly parallel structure, the dissertation proposes adesign idea of fine grain parallelism in pixels of AAM model. This idea makes fulluse of the GPU hardware resources. In order to further improve the algorithm speed,the dissertation proposes a novel auto-tuning algorithm for matrix-vectormultiplication on the GPU, where the number of assigned threads that are used tocompute one element of the result vector can be auto-tuned according to the size ofmatrix. The experimental results show that this algorithm performs quite well for allshapes of matrices. The GPU parallel AAM fitting algorithm is tested on differentdimensions of models. The experimental results indicate that this algorithm achieves aspeedup of dozens of times on high-dimensional texture, and can completely meet therequirements of real-time emotion recognition.4. The dissertation studies the idea and related algorithms of local spatio-temporalfeatures, including the Harris3D spatio-temporal corner detector, the HOG/HOF andthe HOG3D feature descriptors, and the bag-of-words model. On the basis above, according to the dynamics of facial expression, the dissertation proposes the emotionrecognition method based on facial expression local spatio-temporal features,including three algorithms. The first algorithm first adopts face geometrynormalization and Harris3D algorithm to detect spatio-temporal feature points, thenuses HOG/HOF and the HOG3D to describe features, and finally uses thebag-of-words model to classify cognitive-affective states. The second algorithm firstprojects facial texture onto AAM facial standard shape and proposes the facialnormalized texture sequence (FNTS), then adopts HOG/HOF and the HOG3D todescribe features on FNTS, and finally uses SVM to classify cognitive-affective states.The third algorithm improves the second one. It combines the shape features (FNDD)and local texture features, and uses SVM to classify cognitive-affective states. Thesethree algorithms are tested on the dataset. The experimental results show that all ofthem are effective in cognitive-affective states recognition, and the third one achievesthe highest recognition rate which proves the effectiveness of its idea of multi-featurescombination.5. The dissertation analyzes several unsupervised feature extraction algorithmsand studies the stacked convolutional independent subspace analysis (SISA) model.On the basis of the SISA and AAM, the dissertation proposes an algorithm of emotionrecognition based on unsupervised extraction of facial expression spatio-temporalfeatures. This algorithm first adopts AAM to extract facial normalized differencedeformation (FNDD) features and facial normalized texture sequence (FNTS), thenlearn and extract facial expression spatio-temporal features from FNTS, and finallycombines these features with FNDD features on the decision layer and uses SVM toclassify cognitive-affective states. This algorithm is tested on the dataset. Theexperimental results indicate that this algorithm can extract spatio-temporalexpression features efficiently and effectively, and achieves the emotional recognitionrate of more than90%.