Research On Multimodal Sentiment Analysis Method Based On Marker Distribution Learning

Emotion recognition is a hot research area of considerable practical and theoretical interest to a number of fields including machine learning,signal processing and computer vision.Recent facial expression recognition methods usually focus on extracting useful features and applying efficient classifiers such as hidden Markov models(HMM),neural-network-based methods and support vector machine(SVM).Although promising recognition results have been achieved,there still exists a common issue in previous facial expression recognition methods:the assumption that each facial image is associated with only one of the predefined affective labels tends to be an over-simplification.However,they ignore the fact that emotions occur as combinations,mixtures or compounds of the basic emotions and can exist in varying degree of intensity or levels of arousal.In order to solve this problem,researchers apply the label distribution learning(LDL)framework to emotion recognition,and named it emotion distribution learning(EDL).Different from previous researches on emotion classification,LDL framework assigns a set of labels with description degree to an instance,and describes emotions more explicitly.Now,LDL has been able to be applied in emotion recognition skillfully,but it is still a difficult problem to identify emotion more accurately and stably.Therefore,in order to improve the learning performance,this thesis conducts further research on label distribution learning and emotion distribution learning.Firstly,this thesis proposes a label distribution learning algorithm which maintains label ranking relation.Compared with multi-label learning(MLL),the advantages of LDL are reflected in the following quantitative perspectives:(1)the label distribution gives the relevance description of each label to unknown instances;(2)the distribution implicitly gives the relevance intensities relation of different labels to a particular instance,i.e.,the label ranking relation.All existing LDL models aim to fit the ground-truth label distribution,which only uses the first advantage of the label distribution but ignores the label ranking relation,which may lose some useful semantic information implied in the label distribution,thus reducing the performance of LDL.Therefore,this thesis proposes a novel algorithm to solve this problem by introducing the ranking loss function to LDL.In addition,in order to evaluate the LDL algorithms more comprehensively and verify that the ranking loss is beneficial for keeping the label ranking relation,this thesis also introduces a popular ranking evaluation measure for LDL.The experimental results on 13 real-world datasets validate the effectiveness of the proposed method.Secondly,this thesis proposes an emotion distribution learning algorithm based on multimodal information.Existing EDL work has shown a stronger representation ability on emotion recognition,but all are based on facial expressions or other unimodal information.In the real world,it is one-sided to use unimodal information for emotion recognition,and it may produce wrong results.Therefore,in order to make up for the lack of single-modal information dimensions,this thesis introduces multi-modal information into EDL for the first time.Firstly,for each modality,the algorithm learns an emotion distribution and obtains the corresponding label correlation matrix.Secondly,the algorithm constrains the consistency of label correlation matrices between different modalities to utilize the modal complementarity.Finally,the final emotion distribution is achieved based on a simple decision fusion strategy.In addition,as the first pioneering investigation of multi-modal emotion distribution learning,this thesis also extracts the characteristics of multi-modal emotion distribution learning data,which lays the foundation for current and subsequent related research.The experimental results demonstrate that proposal performs better than some state-of-the-art multimodal emotion recognition methods and unimodal emotion distribution learning methods.Thirdly,this thesis achieves an audiovisual emotion recognition system based on multimodal emotion distribution learning algorithm.Automatic emotion recognition systems using facial features or voice features have been explored to a great extent,but the research combining the two methods is relatively rare.A large number of experimental studies show that the combination of video and audio modality will produce better performance.Therefore,we apply multimodal emotion distribution learning algorithm proposed in this paper to achieve an audiovisual emotion recognition system.The system is developed with browser/server architecture,provides visual operations for all stages of human emotion recognition,and can realize the reuse of the modal.