Video Facial Expression Recognition Based On Convolutional Neural Network

Facial expression recognition is an important branch of artificial intelligence and an important means to ensure human-computer friendly interaction and computer intelligence.It has important theoretical research significance and commercial application value.In recent years,a large number of scholars and research institutions have been absorbed into its research.In this thesis,the preprocessing stage of video facical expression recognition is analyzed and studied.The main research works are as follows:(1)Aiming at the problem that neutral expression frames interfere model training and recognition in video sequences,this thesis proposes a new modified method of Dynamic Time Warping,which is named Sliding Window Dynamic Time Warping(SWDTW),to deal with the problem of how to automatically select the distinct facial expression video sequence.And a video facial expression recognition method based on SWDTW and convolutional neural network is proposed.Firstly,face detection algorithm is used to detect the frontal face region of every frame in video sequence.After that,Histogram of Oriented Gradient(HOG)is used to calculate the cost matrix between two sequence frames.Then,the sliding window mechanism is added into the cost matrix,and the global optimal sequence of the expression video is located according to the minimum average distance of all windows.Finally,convolutional neural network is used to classify the selected video frames.The experimental results show that this method can solve the problem that facial expression image is greatly influenced by environmental factors and the problem that the process of traditional feature extraction has too much manual intervention.It can effectively improve the automation and recognition effect of video facial expression analysis.(2)In consideration of the 2D convolutional neural network can not extract the time dimension information,the 3D convolution neural network is applied to the video expression sequence recognition in this thesis.And a video facial expression recognition method which combining the data augmentation and the 3D convolutional neural network is proposed for the sake of solving the problem of the small sample size and the unbalance of classes.In order to expand the difference between expressions,this method first adds Gauss noise to the eye area which is sensitive to the change of expression.Then geometric transformation is applied for the samples which has fewer numbers to solve the sample imbalance problem.Finally,the 3D convolutional neural network is used to carry out unsupervised learning and facial expression classification for the extended facial expression image sequence.The experimental results on the MMI and CK+ expression databases show that the method of data expansion can effectively solve the over fitting problem in the neural network and further improve the effect of facial expression recognition while enhancing the robustness of the model.