| The changes in human internal emotions are usually reflected in the dynamic characteristics of different types of physiological signals.Therefore,individual emotional states can be decoded and identified by integrating multiple modality physiological signals,thus advancing the development of application fields such as mental health assessment and harmonious human-computer interaction.Among the different levels of data fusion,decision-level fusion has the advantage of being insensitive to data asynchronism and modality loss,and can effectively utilize the complementarity between different modalities,which has become one of the main strategies used in multi-modal physiological signal fusion modeling.However,the traditional decision-level fusion method often ignores the correlation between different modalities,which reduces the cooperation ability of multi-modal data and ultimately affects the accuracy of emotion recognition.In contrast,the decision fusion mechanism based on fuzzy integral can consider the interaction relationship of multi-modal data,better represent the correlation between data,and deal with fuzzy or uncertain situations,which effectively makes up for the shortcomings of traditional decision fusion strategies.However,the existing fuzzy measure calculation methods do not comprehensively consider the complex interaction between different modalities,resulting in the weight allocation of different modalities can not reach optimal.Given this,this thesis proposes a new fuzzy measure calculation and optimization method and then constructs a multi-modal fusion emotion recognition model based on the Choquet fuzzy integral,which effectively improves the efficiency of the existing fuzzy integral fusion method.This thesis’ s research content includes:(1)A fuzzy measure calculation method based on fuzzy entropy and association rules is proposed: This method aims to enhance the ability of fuzzy measures to characterize various interactions between modalities,so as to make up for the shortcomings of existing fuzzy measure calculation methods that do not consider the differences between modalities sufficiently.Firstly,this method uses fuzzy entropy as the evaluation index of the importance of a single modality,quantifies the interaction within the modalities,and assigns reasonable weights to different modalities according to their importance,namely,the single-modality measures.Secondly,in order to make full use of the correlation between the modalities,the correlation between the modalities is modeled as association rules,and the single-modality measures are weighted and combined using a lifting process to form the multi-modal measures.Finally,based on the results of the above fuzzy measure calculation,this study uses Choquet fuzzy integral to combine the contributions of different modalities to build a multi-modal physiological signal emotion recognition model.The proposed method was evaluated on three public affective data sets(DEAP,DECAF,and MPED),and the results show that it outperforms two widely used measure calculation methods(De FIMKL algorithm and gλfuzzy measure calculation method)as well as the classical majority vote decision level fusion method in terms of accuracy and robustness.(2)A multi-classifier fusion framework based on Choquet fuzzy integral is proposed: Considering the problem that a single basic classifier is highly sensitive to data from different modalities,this study further constructs different types of basic classifiers for a single modality.Meanwhile,an effective method for integrating decision information of multi-basis classifiers is designed,and a multi-modal and multi-classifier decision fusion framework is constructed by combining fuzzy measure calculation,so as to further improve the overall performance of emotion recognition.Firstly,the framework trains and optimizes several different types of emotional state basic classifiers for each modality,to obtain the decision information of various basic classifiers on multi-modal data.Then,the newly designed multi-base classifier decision information integration method is used to integrate the decision information of each modality respectively,which provides input information for the subsequent calculation of fuzzy measure.Finally,combined with the fuzzy measure calculation method proposed in this thesis,reasonable weights are assigned to each modality,and weighted summation is carried out by using Choquet fuzzy integral to realize the final integration and decision of multi-modal decision information.The experimental results show that compared with the single-classifier fusion model,the multi-modal multi-classifier fusion framework proposed in this thesis can improve the classification accuracy of emotional states by 3%,and also show a better classification effect than the horizontal comparison method.By combining multiple classifiers with different characteristics,the framework can effectively integrate their decision information,make their advantages complement each other,and further improve the effect of emotional state recognition on different data sets and task scenarios.To sum up,the correlation between multi-modal data is deeply studied in this thesis,and the complementarity between different modes is enhanced by optimizing the fuzzy measure calculation method.Based on this,this thesis builds a multi-modal fusion emotion recognition model using Choquet fuzzy integral and further reduces the sensitivity of the model to multi-modal data by introducing the integration strategy of multiple classifiers,effectively improving the performance of emotion recognition,while also providing a new idea and method for the research of multi-modal fusion emotion recognition. |
