Emotion Recognition Research Based On Bimodal Information Fusion

In recent years,with the development of artificial intelligence and deep learning,machine emotion recognition has gradually become a hot topic and has entered people's life.More and more experts and scholars have begun to study human emotions which based on human biological signals,such as human postures,speech signals,expression signals,EEG signals,and ECG signals.Given the convenience of the means of acquisition,voice and expression are the primary means of identification chosen by people.Emotional vector is selected as an emotional model,and there are seven types of emotional states,including happiness,fear,anger,sadness,surprise,suspicion,and neutrality.For the information complementarity of multimodal input data,the emotion recognition which is based on the combination of facial expression and speech is proposed.In this way,the system acquires expression features and speech features respectively from the video sequence and the audio input.It is consisting of frame-by-frame windowing and preemphasis in the step of preprocessing the speech signal.And it is consisting of extracting the rhythm and sound quality characteristics,also including short-term energy,pitch frequency,formant and MFCC,and the statistical emotion parameters were jointly calculated.HMM for Speech Emotion Recognition is applied.For face recognition,an active shape model is established to locate the local position of the face and facial features.Facial expression mainly depends on the texture features of people's faces.Two feature extraction algorithms are proposed:(a)Improved Gabor algorithm,improved Gabor kernel function,making the curvature image more apparent,and increasing the image features of the texture part.(b)The CLBP algorithm that incorporates the Laplacian operator enhances the texture characteristics of the image through the Laplacian operator.Compared with the original feature extraction algorithm,these methods focus on extracting texture detail features of human face.The deep belief network is used for facial expression recognition.Using fusion of decision-making layers,and integrating by product rules,average rules,maximum and minimum rules,result in dual-modal fusion emotion recognition results.The experimental results are evaluated by F-score and G-mean,which verifies the feasibility of the proposed algorithm.