| Given the potential benefits of emotion recognition in understanding and identifying physiological and psychological states of humans,it is frequently used in human-computer interaction to improve user satisfaction and to warn potentially dangerous behavior in advance.Emotion recognition systems typically rely on facial expressions for expression recognition,but these can be influenced by subjective factors,making them somewhat unreliable.EEG signals,which are directly generated by the central nervous system,are less influenced by subjective factors and are therefore important for accurately recognizing human emotions.However,the correlation between the EEG signals of different channels and emotional changes is inconsistent.EMG signals,on the other hand,have certain advantages in identifying negative emotions such as anger,depression,and stress,as these emotions are often accompanied by muscle tension and movement.GSR signals are more suitable for identifying high-intensity emotions such as anxiety,excitement,and happiness,as these emotions can cause changes in skin electrical conductivity.Combining multiple physiological signals can compensate for the limitations of single physiological signal emotion recognition,resulting in improved accuracy and stability of emotion recognition.In this thesis,we conduct research on both single-modal and multi-modal emotion recognition using three types of signals:EEG,EMG,and GSR.We also design and implement a scalable modular emotion recognition system.The main objectives of this thesis are as follows:(1)We propose a multi-channel EEG emotion recognition method based on long shortterm memory networks and ensemble learning.To enhance feature representation,we first study the EEG channels highly related to emotions and select the EEG signals of the temporal and frontal lobes for deep learning model training.During the experiment,the long short-term memory network structure is used to train the selected channel’s EEG signals separately.Ensemble learning strategies such as voting and weighting are used to integrate multiple recognition models and output the final classification result.The experimental results on the DE AP dataset show that the recognition accuracy is improved by using only the temporal and frontal lobe channels compared to using all channels of EEG signals for emotion recognition model training.At the same time,the experimental results show that the deep learning model based on ensemble learning and long short-term memory network is better than the support vector machine model based on feature vector classification.(2)We study multi-modal emotion recognition methods based on EEG,EMG,and GSR signals.Firstly,single-modal emotion recognition based on EMG and GSR signals is studied,and deep learning methods based on CNN-RNN network structure and support vector machine methods are used to train the emotion models of these two physiological signals.Then,feature fusion and decision fusion methods are used for multi-modal emotion recognition research.The experimental results on the DEAP dataset show that the performance of the deep learning fusion model is better than that of the support vector machine fusion model,and multi-modal emotion recognition is more accurate than single-modal emotion recognition.This also confirms that multi-modal emotion recognition can effectively compensate for the shortcomings of single-modal emotion recognition and improve the overall accuracy of emotion recognition.(3)Designed and implemented a scalable and modular emotion recognition system.Based on previous research on single-modal and multi-modal emotion recognition using EEG,EMG,and skin conductance,corresponding emotion recognition models were constructed to validate the system’s scalability.User requirements were determined through requirement analysis,and given the system’s three functional layers of data pre-processing,classification algorithms,and decision fusion methods,which can be freely combined in a tree-like structure,a design was created using the composite design pattern and componentized encapsulation.Ultimately,a fully functional emotion recognition system was realized,which is customizable in terms of data(and its pre-processing),classification algorithms,and decision fusion methods,demonstrating the scalability and expandability of the system’s three degrees of freedom in modal data,classification algorithms,and fusion methods. |
