Research On Facial Expression Recognition Based On Discriminative Regions Boosting And Class Relation Consistency

Emotion is a complex state that integrates people's feelings,thoughts and behaviors,and facial expression is an intuitive reaction of human emotions,which can effectively express individual emotions,cognition and subjective states,and is an important way of interpersonal communication.As a special human behavior analysis task,facial expression recognition can effectively convey non-verbal information and conduct emotional communication in interpersonal communication,so as to help people infer each other's intention.Therefore,facial expression recognition has a broad application prospect in many fields such as human-computer interaction,recommendation system,patient monitoring and so on.Benefitted from the rapid development of deep learning,facial expression recognition has made great progress,but as a result of facial expression inherent variability,complexity,ambiguity and subjectivity,such as facial expression images have the problems of small inter-class differences,large intra-class differences,uncertainty and the noise in the label,the facial expression recognition research has a lot of difficulties and problems to be solved.In view of the above problems,this thesis uses deep learning to solve the variability and complexity problems in facial expression recognition,such as small inter-class differences and large intra-class differences,from the perspective of exploring discriminative areas of the face.From the perspective of introducing the correlation between emotions in semantic space,this thesis explores the correlation between different expressions,and solves the problems of ambiguity and subjectivity in facial expression recognition.The main research contents of this thesis are as follows:1.For the problems of inter-class and intra-class differences,this thesis proposes a joint network based on local and non-local attention mechanism,using local information and global(context)information of facial expressions to realize the feedback of combination optimization of local features and global feature,and automatically search for the key area of facial expression.In this method,U-Net is used to extract the deep semantic information and the comprehensive features of the low-level detail information of the facial expression image.And on this basis,two modules of local multi-network ensemble and non-local attention network are constructed to extract the local and global features of the facial expression image.In the local multi-network ensemble module,local features are extracted by constructing several sub-networks corresponding to local regions of human faces.In the non-local attention network,the weight of global(context)information calculation is input to the local multi-network ensemble module,so as to mine the key area of the face.Experimental results show that the proposed algorithm has better performance than some existing algorithms.At the same time,from the visualization results of the non-local attention mechanism,it can be found that the proposed algorithm can automatically find the key regions of the face.2.Aiming at the inter-class and intra-class differences in facial expression recognition,based on the above method,a more fine-grained label attention mechanism is proposed to mine the key areas of human face.In the deep neural network,the updating of network parameters is determined by the gradient of parameters,and the region with the largest gradient may be the more critical region in the image.On the basis of this assumption,an importance matrix between 0 and 1 is calculated by using the gradient of the network,which is the same size as the input image,and the structure of the original image is destroyed by using the importance matrix,so that the network pays more attention to the key areas in the facial expression image.In order to solve the problem of large intra-class difference,we find the neighborhood for each sample,and amend the feature distribution of the center sample through the features of the sample in the neighborhood,so as to make the distribution within the class more compact.The experimental results show that the label attention mechanism can significantly improve the performance of the baseline model without adding additional training parameters.The visualization experiment results also show that the model can finegrained mine the key areas of the face.The visualization results of feature distribution also prove that the feature distribution within the learned class is more compact.3.In face expression recognition,due to the subjectivity of the labeler and the inherent ambiguity of the expression,most of the existing methods directly assign a probability distribution to each face image to express complex emotions,to solve the problems of uncertainty and noise in face expression recognition task.However,we observed different correlations between emotions,such as surprise and happiness were more likely to occur at the same time than surprise and neutral,which proves that the correlation between different emotions should be taken into account when calculating the label distribution.On this basis,we propose a new method to amend the label distribution of each image by using the correlation between facial expressions in semantic space.Inspired by the correlation between word2 vecs,we first explore the topological information between facial expressions in the semantic space,and embed each image into the semantic space.Then,the semantic correlation between facial expressions was transferred to the task space of facial expression recognition by constructing class relation graph,and the confidence of the label distribution of each sample was evaluated by comparing the semantic class relation graph and task class relation graph of each image.Finally,on the basis of the obtained confidence,the label distribution of each sample is amended by enhancing the sample features with higher confidence.Experimental results show that the proposed method is more effective in constructing label distribution that is closer to people's subjectivity.