Study On Multimedia Information Emotion Recognition Based On Path Signature Feature And Deep Learning Method

Multimedia information emotion recognition technology allows machines to make different feedbacks according to human emotional state,which is of great significance for improving the intelligent level of human-computer interaction.Speech,text,and facial expression signals are the main ways people express emotions,and they contain a wealth of emotional information.Therefore,emotion recognition based on speech and text and facial expression recognition in video are important research directions in the field of multimedia information emotion recognition.Deep learning methods have achieved good results in the field of emotion recognition,but there is still the problem of a large number of complex parameters in the network.Therefore,how to efficiently extract emotional features is still a challenging task.In recent years,as a feature extraction method,path signature has been successfully applied in the fields of handwriting recognition,financial data analysis and other fields,demonstrating its excellent performance in modeling dependencies in series data.In this thesis,the path signature method is used to make targeted improvements based on the characteristics of various data,and a lightweight network is designed to extract effective emotional features with a small amount of parameters.The main work is as follows:1.A lightweight multi-modal emotion recognition network based on high-level feature encoder and multi-branch signature method is designed.(1)For speech emotion recognition network,the high-level feature encoder encodes high-order signature information from the original acoustic features to retain more path local information and avoid feature redundancy;the multi-branch signature method treats the original features and high-level features as individual paths to compute signature features separately,reduce the number of signature features while retaining the emotional information of high-level features.(2)The path signature method is applied to text emotion recognition for the first time,and a text emotion recognition network based on high-level feature encoders is designed.(3)Combine the path signature features extracted by the speech and text emotion signature network to perform multi-modal emotion recognition.The experimental results show that the method in this thesis can effectively extract and combine the emotional features of multiple modal data,and obtain a high accuracy rate with a lightweight network.2.An expression recognition method based on the path signature feature of the landmark trajectory is proposed,and the path signature method is introduced to the task of facial expression recognition in video for the first time.The method includes using the path signature feature to characterize the dynamic change information of the landmark trajectory,designing a lightweight network to extract the spatial structure feature,and introducing the signature feature attention module.In this way,it can extract effective spatio-temporal features without introducing training parameters,make full use of facial structure information,and make the network focus on signature features that are more related to emotions.Experimental results show that the method proposed in this thesis achieves an accuracy comparable to the most advanced algorithms with very few parameters,and is superior to other methods based on landmark trajectories.