Near-infrared Facial Expression Recognition Based On Convolutional Neural Network With Automatic Assignment Of Feature Weights

Facial expressions contain abundant non-verbal information.Machines that are able to understand facial expressions can better serve humans and fundamentally change the way of interaction between humans and machines.Facial expression recognition is widely used in various fields of society,such as virtual reality,human-computer interaction and security.However,facial expression recognition system is easily affected by many factors.Firstly,facial expression recognition tends to be affected by illumination changes.Although some pre-processing algorithms can reduce the impact of illumination changes,the effect of processing is limited,and even some algorithms will cause some loss of effective information.Secondly,static images of facial expression only include appearance features and cannot include dynamic features.Finally,not all facial information is useful for facial expression recognition.It is unknown that which part of the face should assigned more weight for facial expression recognition.Because the 3D convolutional neural network(3D CNN)can extract the dynamic features of the facial expression sequence,and the infrared expression image is not affected by the indoor illumination changes.In response to the above problems,we propose a three stream Fusion Network with Squeeze-and-Excitation(SE)block called SETFNet(Three stream Fusion Network with SE block)for near infrared(NIR)facial expression recognition.We used three different local parts of the face,namely eyes(including eyebrows),nose and mouth,as input to three streams of the network,which provides accurate expression features while the smaller pictures are less computationally intensive.In addition,we embed an SE module that can adaptively learn feature weights in the network,so that the network can enhance the effective features,weaken the useless features,thereby improve the recognition rate.The SETFNet proposed in this paper is evaluated and tested on Oulu-CASIA database.The experimental results show that SETFNet can adaptively learn the weights of feature channels through the network,enhance the useful features for the tasks and weaken the useless features.In the case of using only local face information,it can achieve higher recognition rate than most of algorithms at present,which proves the effectiveness of the method proposed.