Research On Multi-modal Emotion Recognition Combining Face Image And EEG Signal

With the increasing development of artificial intelligence,human-computer interaction is becoming more and more important in real life,and the realization of good humancomputer interaction is inseparable from the recognition of emotions.In addition,emotion recognition also has a wide range of applications in medical and health,national defense and security,commercial finance and other fields.At this stage,emotion recognition can be divided into emotion recognition based on physiological signals and emotion recognition based on nonphysiological signals.Among them,emotion recognition based on human faces is an important part of emotion recognition based on non-physiological signals.Face-based emotion recognition has the advantages of convenience and speed,but in the case of deliberately hidden subjective,it will cause inaccurate recognition;the use of physiological signal recognition can effectively prevent the occurrence of the above problems and recognize the real human emotions.In response to the above problems,this paper uses a one-dimensional spatiotemporal neural network to perform emotion recognition on EEG,uses a deep separable residual convolution network to perform emotion recognition on face image data,and finally combines the recognition results of the two in the decision-making layer.The specific research content of this article is as follows:(1)Research on EEG emotion recognition based on one-dimensional spatio-temporal neural network.Describes common emotion EEG data collection,classification and calculation methods introduced parametric feature common EEG emotional,functional and principles cycle neural networks were introduced.Then,the single convolutional neural network and the single recurrent neural network can only extract space or time features respectively,so the recognition rate is not high.For the one-dimensional time series signal of EEG,the onedimensional convolution replaces the traditional two-dimensional volume.Product,to improve the multi-layer convolution of the local path,after passing through the multi-layer onedimensional convolution network,connect two layers of long and short-term memory networks and then connect to the fully connected layer,and design a one-dimensional spatio-temporal neural network structure by combining Conv1 D and Conv LSTM,Can extract the time dimension and space dimension features at the same time,and optimize the EEG data source.Before EEG recognition,according to the human brain area,the relationship between EEG signal and emotion,the EEG level is first selected,reducing irrelevant information and The amount of calculation,and finally through the classification experiment on the DEAP EEG data set and the kaggle EEG emotion data set,the recognition rates can reach 78.14% and 99.08%,respectively,which proves that the improved model can significantly improve the classification accuracy and better extract The features in the EEG signal can achieve emotion classification more accurately.(2)Research on facial emotion based on deep separable residual convolutional network.First,the basic structure and principle of the basic residual network and the depth separable convolutional neural network are introduced,and then the gradient disappears caused by the ordinary convolutional neural network with the deepening of the network and the increase of the parameter amount,the degradation of the gradient explosion network and the parameters The problem of difficulty in training caused by excessive amounts,a deep separable residual convolutional network is proposed.The framework adds residual blocks between the separable convolutional layer and the two-dimensional convolutional layer.The depth of the separable convolutional network is greatly The number of parameters of the network is reduced and the training efficiency is improved.The residual network introduced on this basis ensures that there will be no problems such as network degradation while deepening the network.The accuracy rate on the FER2013 data set reaches 70.89%.Experiments prove the effectiveness of the constructed deep separable residual convolutional neural network,which can improve the emotion recognition rate and achieve better results.(3)Emotion recognition combining one-dimensional spatio-temporal neural network and deep separable residual convolutional network.A bimodal decision-making fusion emotion recognition model architecture is proposed.Since single-modal emotion recognition is easily affected by the characteristics of the modal itself,DS decision fusion uses certain synthetic rules and decision methods to combine the results of Chapter 3 and 4 together,and combine the two classifiers to construct An emotion recognition framework that integrates EEG information and facial visual information.The experiment is carried out on the DEAP emotion database.The experiment shows that compared with a single modal,the proposed bimodal emotion recognition framework has better performance and can integrate people well.The emotional information contained in facial visual signals and EEG signals has certain application value in real life.