Facial Expression Recognition Methods Based On Deep Learning

Deep learning based methods have achieved great success in many domains including natural image applications,natural language applications,face applications,etc.It grabs the attention of academia and industry.The deep learning methods include a series of multi-layered nonlinear pattern recognition systems.These systems extract abstract features layer by layer.Compared to the shallow models,deep models are suitable for handling the highly nonlinear problem in the real world.Facial expression recognition is a typical real-world highly nonlinear problem.This paper focus on the central task of facial expression recognition,explores deep learning algorithm-s include 2D Convolutional Neural Networks,3D Convolutional Neural Networks,Binarized Neural Networks,Binarized Auto-encoders,Convolutional Neural Networks with Visual At-tention,Deconvolutional Neural Network,etc.The main contributions of this work can be summarized as follows:(1)This paper proposes a facial expression recognition method based on Convolutional Neu-ral Networks and a convolutional feature transfer method.Although the methods obtain good performance,its decisions are unexplainable.Psychologists explain the facial ex-pressions as combinations of facial action units.The action units are located in a specific region of interests.To explore these regions,improve the facial expression recognition rate,and confirm the psychological definitions,this paper proposes a visual attention based facial expression recognition method.The visual attention model detects the AU-aware regions,amplifies the signals in the regions,suppre,sses the background noises,and improves the recognition rate.(2)A well designed facial expression recognition system should be sensitive to expressions,and invariant to identifications.To explore the expression components and the identi-fications component in facial representations.This paper proposes the NET-3 system containing a pair of 18-layered Convolutional Deconvolutional Neural Networks.The main purpose of the NET-3 is to extract complementary facial representations.The ex-tracted complementary facial representations are used to reconstruct the original faces,generate new faces,interpolate new faces,facial expression recognition,and facial ver-ification.To meet the experiment requirements,a new facial expression dataset called Large-scale Synthesized Facial Expression Dataset is presented.It is the largest facial expression dataset among the existing ones.(3)Most facial expression methods make decisions according to the facial appearance.The facial actions based facial expression recognition methods can be more accurate than the appearance based methods.To make full use of the abundant action information in the video data.This paper further studies on the 3D Convolutional Neural Networks based fa-cial expression recognition method,defines a 3D Convolutional Neural Networks frame-work,and discusses the basic rules of designing 3D Convolutional Neural Networks.To build a high-performance recognition system,3D Gabor feature and optical flow feature is employed.Four 3D Convolutional Neural Networks,namely 3DCNN-A,3DCNN-B,3DCNN-C,and 3DCNN-D,are designed.Finally,the decisions based on different features and networks are fused together.High accuracy is obtained.(4)The memory costs and computation costs of mainstream deep learning algorithms are usually higher than those of shallow methods.To lower these costs of large-scale deep neural networks and make the real-time propagation faster,this paper studies on an ef-ficient facial expression recognition system based on Multi-scale Dense Local Binary Patterns,Stacked Binarized Auto-encoders,and Binarized Neural Networks.The system achieves good performance with minimal hardware requirements,i.e.,lower memory and computation costs.