Research On Multimodal Emotion Recognition Technology

People hope that computers can have their own thinking and consciousness like humans,recognize human emotional states,and communicate with humans very naturally.Because the emotional information carried by single-mode emotional signals is relatively limited,and single-mode emotional recognition has some defects,such as low accuracy and poor robustness,The emotion recognition method of dual-modal fusion or multi-modal fusion has increasingly become the focus of researchers.The effective complementarity of emotion information between multiple modes can significantly improve the performance of emotion recognition.In this paper,text mode,voice mode and video mode are used for multimodal emotion recognition,and the related methods of machine learning and deep learning are used to study multimodal emotion recognition.In this paper,the two-way long-term and short-term memory network is used for multimodal emotion recognition.The interdependent emotional relationship is established before and after each emotional state,and the emotional information inside and between each mode is fully explored.In the process of emotion recognition,the multi-modal emotion recognition method based on feature layer fusion can effectively give play to the significant advantages of information complementarity between different modes,capture the emotional information and emotional connection between different modes,make up for the defect of low performance of single-mode emotion recognition,and improve the performance of emotion recognition.However,the way of feature level fusion is to integrate the emotional features of each emotional mode,which inevitably leads to the problem of high emotional feature dimension.Aiming at the problem of high emotional feature dimension caused by feature level fusion,this paper uses the method of kernel principal component analysis to reduce the dimension of emotional feature,for each emotion mode,the weight of each emotion mode is allocated to maximize the contribution rate of each emotion mode.In the process of emotion recognition,the multi-modal emotion recognition method fused by the decision-making level can give full play to the unique advantages of each mode,but the emotional features extracted by each single mode will have different degrees of emotional feature redundancy.These redundant emotional features will not only increase the computational consumption of emotion recognition,but also reduce the performance of emotion recognition,In this paper,the maximum information coefficient method is used for feature selection.Through feature selection,redundant emotional features and irrelevant emotional features can be removed,and the accuracy of multimodal emotion recognition can be significantly improved.The rules of weighted summation and voting are used as the decision rules for decision-making level fusion.Both feature level fusion and decision level fusion have irreplaceable advantages.In order to give full play to their respective advantages,this paper adopts the multi-modal emotion recognition method of hybrid layer fusion,and secondary fuses the emotion recognition results of feature level fusion with the emotion recognition results of decision level fusion,which can effectively improve the performance of multi-modal emotion recognition.