Researches Of Dual-modality Emotion Recognition Based On Temporal-spatial Features

The understanding and expression of emotion is an important means of human communication activities.With the rapid development of artificial intelligence and other fields,intelligent human-computer interaction should meet the demand of "natural" interaction between users and machines,while the key to realize "natural" interaction is to let the machine have the ability to simulate human emotion cognition and expression.Visual emotion information which can be seen as the most intuitive form of emotion expression has important significance in the research of affective recognition.In this paper,based on the visual emotion information,the dual-modality emotion recognition is realized by the fusion of facial expression and upper body posture features.The main contents and innovations of this paper are as follows:(1)Video is stacked by image frame sequence according to the order of time and space.In this paper,the variety of facial expression and upper body posture in the video is considered as three-dimensional space-time motion.And a method of dual-modality emotion recognition based on facial expression and upper body posture is proposed,which can overcome the limitations of single modal emotion recognition and get more reliable emotion recognition results.(2)In order to solve the problem of high computational complexity and low recognition accuracy in video emotion recognition,we propose a feature description method named Temporal-Spatial Local Binary Pattern Moment by improving the existing algorithms.Firstly,the K-means clustering algorithm is used to obtain the image sequences of facial expression and upper body posture.Secondly,the expression and posture images are divided into blocks separately.Temporal-Spatial Local Binary Pattern Moment algorithm is used to get the emotion feature of the image block.The Euclidean distance between feature vector of the test sample and each training sample is calculated,and the minimum distance of each emotion category is used as evidence to construct the basic probability assignment value.Finally,the final result is obtained by fusion of two modal information using the D-S evidence theory.The experimental results show that,compared with Volume Local Binary Pattern algorithm,Temporal-Spatial Local Binary Pattern Moment algorithm has the advantages of low feature dimension,strong representation ability,and good recognition effect.(3)Aiming to overcome the limitation of Temporal-Spatial Local Binary Pattern Moment algorithm in the expression of local texture details,a new method is proposed.On the basis of Temporal-Spatial Local Binary Pattern Moment algorithm,we add the adaptive threshold,do three value quantitative calculation,and put forward Temporal-Spatial Local Three Pattern Moment algorithm.The experimental results show that Temporal-Spatial Local Three Pattern Moment algorithm describes image texture more delicately and accurately than Temporal-Spatial Local Binary Pattern Moment.In addition,in order to avoid the inadequacy of single feature expression ability,Three-Dimensional Histograms Oriented Gradients,which can describe the image emotion from the edge and direction,is joined to form a composite spatio-temporal feature with Temporal-Spatial Local Three Pattern Moment.The composite spatio-temporal feature can effectively enhance the ability of expressing the emotion information for video,which is better than two single algorithms and other methods.At the same time,the D-S evidence theory which fuses the information improves the recognition performance of the system.