A Study On Facial Expression Recognition Based On Image Sequences

With the promotion of the biometric identification technology,facial expression recognition has gradually emerged.Whether in academic or industrial,automatic facial expression recognition has been an active research topic.It has enormous potential for development and can be used in intelligent human-computer interaction,mass entertain-ment,safe driving,medical assistance,online education,etc.So far,facial expression recognition algorithms mostly use static facial expression images as research objects.However,the static image is difficult to reflect the dynamic characteristics of the expres-sions,which hinders the development of facial expression recognition to some extent.In contrast,facial expression image sequences can provide richer texture and motion information about expression changes,and it can significantly improve the accuracy of facial expression recognition if used properly.Therefore,facial expression recognition based on image sequences has significancant research value.The main work and contributions of this dissertation include the following aspects:1.Face detection is studied deeply.Due to traditional face detection methods only detect frontal or close-to-front faces,we make some improvements.Specifically,based on the multi-view face detection algorithm developed by Yahoo,we improve the face detection effect and network training speed of the algorithm by changing its network structure and establishing a more appropriate face database.2.A dynamic facial expression recognition method based on hybrid features is pro-posed.First,two novel dynamic features are constructed based on the extracted facial landmarks:dynamic geometric feature and dynamic texture feature.The dynamic ge-ometric feature not only uses the motion direction of facial landmarks,but also makes use of the shape changes from the eyebrow,eyes and mouth regions.Second,LBP features are extracted from neighborhood regions centered on facial landmarks,and the normalized cross correlation coefficients of LBP features between pairwise images is utilized as a representation of local texture differences of the faces.Integration of both geometric and texture features further enhances the representation of facial expressions.Through a large number of experiments on CK+ database,the results show that the al-gorithm can achieve a competitive performance.3.A dynamic facial expression recognition algorithm using convolutional neural networks is presented.It selects the LBP images as the inputs of designed network structure.First,the network structure is finetuned on FER2013 database.Then,the features of the images are extracted under the trained network model.Next,in one image sequence,the mean,variance,maximum and minimum of feature vectors over all frames are calculated according to its dimensions and combined into a vector as the feature.Finally,SVM is used for facial expression recognition.The performance of the algorithm is evaluated on AFEW database,the results verify the effectiveness of the algorithm.