Facial Expression Recognition Based On CS-LOP And Convolutional Neural Network

In recent years,facial expression recognition as a branch of pattern recognition and affective computing has attracted many researchers.At present,the traditional expression recognition mainly focuses on feature extraction,but some handcrafted features cannot meet the requirements of accuracy due to shortcomings such as low robustness,high time complexity,and weak feature representation ability.Therefore,this thesis is devoted to designing local feature description operators with good representation capabilities and strong robustness to improve the accuracy of facial expression recognition.It is worth mentioning that deep learning has been favored by many researchers due to its excellent learning ability and good recognition accuracy.Therefore,this thesis combines handcrafted facial features with deep learning for facial expression recognition.The main work is as follows:(1)It summarizes the background,improtance and current status of facial expression recognition,and introduces the current widely used facial expression dataset,then analyzes the difficulties in the research of this subject.This thesis describes each process in the facial expression recognition system in detail,mainly including face detection and preprocessing,feature extraction,classification recognition.We also introduces the theory of deep learning in facial expression recognition systematically.(2)Aiming at the traditional local texture feature descriptors ignoring the relationship between the gray values of neighboring pixels in all directions,this thesis focuses on LBP and CS-LBP,proposes a new texture descriptors: Center-Symmetric Local Octonary Pattern(CS-LOP).CS-LOP expands the traditional two-bit binary code to three bits,which increases the coding range and depicts more detailed texture features.Meanwhile,to improve the accuracy of expression recognition,not only the Gabor features and gradient magnitude features of the original facial images are extracted,but also the CS-LOP features of Gabor feature map and gradient magnitude feature map.Then,the above features are fused to achieve expression classification.The experimental results on JAFFE and CK databases illustrate the effectiveness of this method.(3)Due to the limited promotion effect of handcrafted features and the need of manual selection on expression recognition and considering the excellent results of deep neural networks in expression recognition,this thesis proposes a framework which combined CS-LOP and Convolution Neural Network(CNN)method for facial expression recognition.On one hand,the extracted handcrafted feature CS-LOP focus on the salient texture features of the local image,so that the indistinguishable differences between pixels can be characterized.On the other hand,the CNN as an endto-end learning method achieves image features extraction and classification automatically,and the stacking of multi-layer neural networks makes it easier to extract the deep features of images.In addition,this thesis introduces the idea of multi-task learning,taking the original expression image as the main task,and the region of interest with a large expression contribution as the auxiliary task.Through the joint training and weight sharing of CNN,the auxiliary task continuously revises the results of the main task.Then,the CNN and CS-LOP features are used to identify the expression categories in parallel.These two branches obtain final recognition result through a voting mechanism.The experimental results show the feasibility of the method proposed in this thesis.