The Research Of Multi-Modal Fusing Based Emotion Recognition

Emotion recognition is a challenging task in affective compute,due to the difficulty of discriminative feature extraction from audio-visual data,which utilizes three types of strategy to fuse the audio and visual feature,i.e.,feature level fusing,decision level fusing and modal level fusing.The feature level fusing and decision level fusing strategies are shallow fusing strategies that conduct a lack of discriminant of the joint feature.However,a well-designed fusing modle or a neural network costs too much time to stract the joint feature because of the complicated design and the vast parameters.This thesis,aiming to solving the contradiction between lack of feature discriminant and time consumption of feature extraction,proposes a deep level modal fusing network to implement multimodal emotion recognition.The audio-visual data is first cut into segments with sliding windows,where each segment contains a key picture and a chunk of speech signal.Human face is cropped with bounding box and speech signal is transformed into 3-dimention Mel-Freqency Cepstral Coefficients feature map.These two data streams are fed for the network in first two parts to extract feature and be fused into a joint representation by the depth wise separable convolution with residual layer.The fully connected neural network is used at last to train the classifier of seven human emotions,including happy,sad,suprise,disgust,angry,fear and normal.Experimental results conducted on the RML,the acted eNTERFACE05,and the BAUM-1s dataset yield the state of art performance.To expand the applications of emotion recognition technology in a various fields,this thesis designs and implements a multimodal fused emotion recognition and analysis system,which consists of six modules:data collecting,data processing,emotion recognition algorithm,data analysis,data visualization and system manager.The system is capable to conduct both online and offline emotion recognition tasks,depending on their different data modals,file types,and recognition algorithms accordingly.This thesis gives a detailed design and implementation of the system with all the function parts and performance tested.It is also applied for view analysis in social media and video assistant for medical rehabilitation.