3D Facial Feature Reconstruction And Learning Network For Facial Expression Recognition In The Wild

As facial expression recognition(FER)is one of the most important forms for human conveying their emotions and states,it has been an indispensable part of artificial intelligence and one of the most popular fields of computer vision.Although computer vision and robotics have made great progress since the wave of deep learning,some problems in FER are still remained that needed to be solved.For decades,the establishment of various facial expression databases explicitely has drived the great development of automatic FER.The facial expression databases can be roughly divided into two categories,one is collected from laboratory,the other is collected from real-world or unconstrained conditions.Although facial expression recognition model based on deep learning technology have achieved perfect recognition accuracy and exceeded previous results by a large margin on the laboratory expression databases,the performance of these model dramatically deteriorates when applied to FER in the wild.The reason is that the images about facial expression databases in the wild are collected from movies or Internet,and they are commonly influenced by factors such as posture,occlusion and illumination.These factors in unconstrained scenarios are not only the disadvantages of FER,but also the directions of researchers devoting to tackle.The FER in the wild is the only way for expression recognition being intelligent.The proposed method in this thesis based on 3D face feature reconstruction and gragh learning improve FER in the wild accuracy and robustness.Specifically,the research contents of this thesis are as follows:(1)The application of 3D feature reconstruction and deep graph learning in facial image analysis is introduced,and several popular algorithems of these two technologies are listed.Then the feasibility of applying these two technologies to facial expression recognition is analyzed and the corresponding application scheme is proposed.(2)The commonly used algorithms for FER in the wild are investigated,and some of them,which can be divided into three categories: posture problem-oriented methods,occlusion problemoriented methods,and multiple problem-oriented methods,are reproduced.Comparing the experiment results of each of them,the strengths and weaknesses are analyzed.(3)We proposed a novel end-to-end trainable deep neural network model based on 3D facial feature reconstruction to alleviate the challenges in facial expression recognition in the wild,named3 D facial apparent and geometric feature reconstruction and learning network(3DF-RLN).3DF-RLN is mainly consisted of four modules: 1)3D face reconstruction;2)the appearance pathway;3)the geometry pathway and 4)fusion and recognition.(4)At first,3D face reconstruction is used to generate facial apparent features and facial geometric feature.Then convolutional neural network(CNN)and graph convolutional network(GCN)is used to extract facial information from apparent feature and geometric feature seperately.Finally,the fusion and recognition module is used to fuse the expression features from two pathway to achieve FER in the wild.It is worth mentioning that,using the trainable adjacency matrix of the GCN in the geometry pathway,our method can not only achieve FER in the wild,but also obtain the interconnection relationship and importance between each facial landmark.(5)Extensive experiments are conducted on 3 benchmark databases: Multi-PIE,RAF-DB and Affect Net.The results show that the proposed method can achieve better results than the related stateof-the-art methods of FER in the wild.