Research On Multi-modal Emotion Recognition Based On Broad Learning System

Emotion recognition is a hot and difficult topic in artificial intelligence in recent years.People's emotions can be expressed through facial expressions,voice,body postures and other different modes.At present,here are many researches on emotion classification related to static facial expressions and voice,but there are relatively few researches on multi-modal emotion recognition based on facial expressions and head postures.This paper studies the emotional expression characteristics of facial expressions and head postures,and establishes an emotion recognition model based on these two modes using the Broad Learning System(BLS)algorithm and verifies it on the public emotional video dataset.When facial expression information is extracted,the second full connection layer of 3D convolution network is used as the facial expressions input,and a 2048 dimensional feature is extracted from each video sample.For the head postures,this paper uses the method of facial feature point structure localization to solve the three-dimensional Euler Angle of the head to represent its postures.For each video sample,a 33 dimensional emotion feature of head posture is extracted.Before bi-modal fusion,we trained the single modal broad neural network model of facial expression and head posture.Among them,the single modal broad neural network model of facial expression has achieved 93.9% classification accuracy on the traditional CK+ expression database,but only 41.1% accuracy on the CHEAVD2.0 video emotion database.The accuracy of the head single modal broad neural network model on the CHEAVD2.0 video emotion database is 33.5%.From the experimental results,it can be seen that the single modal classifier has a general effect on the task of emotion recognition in complex real scenes.In order to improve the recognition effect,we use the broad neural network to perform feature layer fusion and decision layer fusion for facial expression and head posture respectively.The features of facial expression and head posture were processed by using double broad neural network structure in feature layer fusion,and then the emotion classification model was established by BLS.Decision layer fusion is that the output of front two single modal broad neural network model about probability distribution are trained again.Then,we get a decision fusion model based on the broad neural network,and the accuracy of decision fusion model on the CHEAVD2.0 validation set is 42.3%,which better than the classification accuracy of single modal emotion recognition model.