Research On Emotion Classification Based On Multi-moda Data

With the development of online video networks,such as Tiktok,micro-blog and YouTube,more and more people are inclined to share videos to express their feelings or opinions on hit events.Mining valuable emotion information from it has great impacts on the study of real-life applications,such as box office prediction,public opinion monitoring,intelligent customer service and product recommendation systems.However,unlike the text emotion analysis,this paper conducts emotion analysis on multi-modal data,such as videos and audios.The additional modalities,such as visual and acoustic modalities,make this task more challenging.For instance,the pause and mood phrases in utterances,the change of facial expressions and the switch of tone and volume in verbal voice.The core challenge of multi-modal emotion analysis is how to model the intra-modal dynamics and the inter-modal dynamics more effectively,which draws much more attention from aca-demic and industrial communities.This paper is about multi-modal emotion classification analysis,mainly focusing on the problem of multi-modal data modeling.Detailed re-searches are as follows:Firstly,this paper proposes a multi-modal emotion classification approach based on multi-task learning in a scenario where the speaker in the experimental corpus may express several emotions in one utterance.The main idea of this approach is to transform the mul-ti-label emotion classification task into multiple binary classification tasks and simultane-ously learn these tasks through the multi-task learning framework,so as to capture the rela-tionship among different emotion categories.The private and shared network layers are built to model the intra-modal and inter-modal dynamics to obtain the final multi-modal emotion representation.Experimental results demonstrate our approach can efficiently boost the performance of multi-modal emotion classification.Secondly,this paper proposes an approach to multi-modal emotion classification based on auxiliary sentiment information.Previous studies on text emotion analysis show that the performances of emotion and sentiment classification tasks can be effectively im-proved by learning sentiment and emotion information together.In order to learn the rela-tionship between emotion labels and sentiment labels from the multi-modal data,this ap-proach uses a joint learning framework where the multi-modal sentiment classification task(the auxiliary task)can assist the learning of multi-modal emotion classification task(the main task).After capturing the auxiliary textual and acoustic representations of the main task,they are integrated with the original information from the main task so as to enhance its ability to perform.Experimental results demonstrate the proposed approach can effec-tively improve the performance of multi-modal emotion classification.Finally,this paper proposes an approach to multi-modal emotion classification based on non-redundant information learning.Normally,there may be redundant information when modeling the multi-modal interactions.Besides,different modalities in an utterance may express different emotions.Therefore,the main idea of the proposed approach is to distinguish the intra-modal and inter-modal dynamics with the orthogonal constraints so as to remove the redundant.Furthermore,the gated fusion mechanism is employed to help control the contribution of each modality to obtain the final multi-modal emotion repre-sentation.Experimental results demonstrate the effectiveness of our proposed approach to multi-modal emotion classification.