Researches On Data-driven Emotion Recognition Algorithm

Emotions play an important role in human communication,which can reflect the psychological or physiological states of human beings and drive individual behavior and decisionmaking.In the field of artificial intelligence,it is important to understand human emotions,and to give machines affective cognition to serve people in a more intelligent and friendly way.For example,emotion recognition can help machines interact with humans more user-friendly,assist in medical diagnosis to determine patient status,and enhance player experience in entertainment games,etc.Human emotions can be conveyed through physiological and non-physiological signals,and the appropriate signals can be used for emotion analysis according to the needs of application scenarios.The work is driven by data types and investigates emotion recognition algorithms for different data scenarios,based on the characteristics of electroencephalogram(EEG)data and facial image data,and the main contributions are as follows:(1)EEG emotion recognition based on topology structure.The spatial structure of EEG data is non-Euclidean,which can be modeled by graph networks,but the performance may be degraded by pattern collapse as the depth of the network increases.To address this problem,a deeper and wider graph network,GCB-Net,is proposed to learn the high-level emotion features of EEG data.The shallow graph convolutional neural network(GCNN)can capture the correlation between EEG channels,and the subsequent convolutional neural network(CNN)can learn higher-level features.The broad connection concatenates the shallow and deep layer features,providing multi-level features for emotion recognition.In addition,experimental results on the EEG emotion recognition dataset demonstrate the good performance of GCB-Net.(2)Expression recognition based on facial region actions.Facial expressions can be regarded as a combination of multiple facial region actions.However,it is not accessible to obtain the action information of the region by dividing the facial image into patches.Therefore,in this paper,we construct a multi-stream region network RAU-Net by combining action units(AU),which divides the image into several small regions,and learns AU features of each region separately,and then integrates them for final expression classification.The experimental results show that the proposed RAU-Net can learn discriminative regional action features,which can improve expression recognition performance.(3)Expression recognition based on privileged action information.Given the limited information provided by a single image,AU information can be used to enhance the expression recognition model.However,the direct use of AU information increases the network complexity,so this paper adopts privileged learning to implicitly integrate AU information into the deep expression recognition network without increasing the computation in the inference process.The privileged action unit network(PAU-Net)has two ways for information to flow into the network: AU information as privileged input(Type I)and privileged output(Type II).Type I uses AU as input and utilizes the emotion labels predicted by AUs as auxiliary labels to constrain the learning of the expression network.Type II uses AUs as the learning label for the shallow local network,which enables the network to learn AU-related features at the shallow layer and the deeper network further learns more discriminative expression-related features.In this paper,experimental validation is conducted on several expression recognition datasets,and the results show the effectiveness of PAU-Net.(4)AU recognition based on facial semantic learning.AU describes the way of muscle movement in the facial region,which is closely related to facial morphology.Therefore,the AU recognition task can benefit from understanding the semantic partition of the face.Conventional methods usually rely on facial landmarks to obtain facial morphology information,and the model itself can not understand it.Driven by this observation,this paper proposes SEi T,a Transformer based model with facial semantic embedding,which can learn facial semantic features,and interact with all region semantic embeddings.Experiments show that facial semantic learning can improve model performance,and the proposed SEi T achieves outstanding results on AU recognition datasets.