Facial Expression Recognition Based On Deep Geometric Features

Facial expression recognition has profound meaning and broad application value in various fields of artificial intelligence.It has been a hot research issue in affective computing and human-computer interaction.Expression recognition technology can use computer vision methods to automatically recognize the facial expression from images or videos and then perceive the emotions of people.According to different research objects,facial expression recognition can be classified as static macroexpreesion,dynamic macro-expression and micro-expression recognition.In recent years,deep learning has achieved remarkable research findings in computer vision.Various deep learning models have been proposed and boost the performance of facial expression recognition.Facial geometric features can accurately depict the trajectory of facial features in the movement of facial expressions.However,existing work focuses on extracting facial apperance features from images or videos,lacks robust and efficient extraction methods for facial geometric features.This thesis focuses on deep geometric features and conducts research on static macro-expression,dynamic macro-expression and micro expression recognition.Firstly,to solve the problem that static macro-facial expression recognition only focuses on single feature extraction,a Multi-feature fusion method is proposed.This method uses geometric features as a complement to image features.In terms of image features,an image feature extractor is designed which can efficiently extract facial texture features from images.In terms of geometric features,we perform distance transformation and feature extraction on facial landmarks to extract robust and discriminative deep geometric features.Finally,we use weighted summation and jointly training tofuse image features and geometric features and output the classification results.The experimental results show that the multi-feature fusion method outperforms existing static macro-expression recognition methods by considering both geometric and image features.Secondly,to solve the problem of effectively using video and geometric spatiotemporal features in dynamic macro expression recognition,a spatio-temporal feature fusion is proposed.In terms of spatio-temporal features of the video,we regard each frame in the video as a static macro-expression recognition sub-problem,extract the spatial features from the static frames and then use the temporal feature classifier to establish the temporal relationship between the spatial features,which is better for capturing the facial movements.In terms of geometric spatio-temporal features,we perform spatio-temporal distance transformation and spatial grouping on facial landmarks to improve the representation and discrimination of spatio-temporal geometric features.Finally,we fuse the spatio-temporal video and geometric features and output the final classification result.Experimental results show that the method of fusing spatio-temporal features achieves better recognition accuracy.In addition,compared with existing methods such as 3D convolutional neural network and step-wise spatiotemporal feature extraction,our method also has the advantage of fewer parameters,less training time,variable-length input available and the end-to-end model architecture.Finally,to solve the difficulty of extracting effective geometric features in microexpression recognition,a saliency region based method is proposed.Micro-expression only occurs in certain facial areas,compared to macro-expressions,it is more necessary to effectively use geometric features.However,due to the small action amplitude of the micro-expression,the facial landmarks are not accurate enough to represent the geometric features.Therefore,we convert facial landmarks into saliency regions as the geometric features.During the process of recognition,the model focuses on the texture variation of salient regions to better extract the tiny expression actions.Definitely,this method first defines the concept of saliency regions as geometric features and label transformation method,then trains the saliency region generator to automatically locate the saliency region and output corresponding heatmaps as geometric features.These geometric features are used as attention weight to active or de-active corresponding neurons during recognition,so that the model pays more attention to saliency regionsindicated by heatmaps and eliminates the noises caused by the unrelated regions.Experimental results show that saliency region based method outperforms existing works on differnet metrics and extracts more discriminative features.