Research On Facial Expression Recognition Based On The Fusion Of Visible Images And Infrared Images

Facial expression recognition belongs to a cross-disciplinary subject,involving the computer science,image processing,pattern recognition,computer vision,human-computer interaction,psychology,physiology and fields of other disciplines.Expression recognition research not only can improve the theoretical system of emotional computing,but also promote the development of harmonious human-computer interaction.At present,the facial expression recognition algorithms are mainly concentrated in that it is easily influenced by illumination change under the visible light environment,and the infrared human face image reflects the temperature distribution on the surface of the face,which has strong robustness to illumination,to a certain extent,the infrared human face image can make up for the inadequacy of visible light image.As the surface texture of infrared images is not clear,the research of infrared facial expression recognition has been restricted.Considering the characteristics of visible face image and infrared face image,this paper studies and discusses the fusion method of visible face features and infrared face features.It aims to enhance the robustness of facial expression recognition to illumination and improve the performance of facial expression.The main work and innovation of this paper are as follows:1.In this paper,the principle and feature extraction method of LBP feature,LDP feature and LPQ feature are introduced,and the classification algorithm of support vector machine is described in detail.The facial expression images containing both visible and infrared images were selected as the sample images in the USTC-NVIE expression database.The expressions are divided into three types:positive expression,negative expression and neutral expression.The LBP features,the LDP features and LPQ features from visible light image and infrared image are extracted respectively,and three kinds of facial expression recognition experiment is carried out based on support vector machine(SVM)classification method.The experimental results show that the accuracy of visible light expression recognition is much higher than that of infrared expression recognition,while the recognition rate based on LBP features is higher than that of the other two features.2.A smile recognition method fused of visible light images and infrared images based on the contrast pyramid decomposition is proposed in this chapter.Firstly,seven feature points are calibrated from the visible face image and the infrared face image respectively,and the visible light feature points and the infrared feature points are matched one by one to conduct the image registration of visible image and the infrared image;And then fuse the visible and infrared facial images based on the contrast pyramid decomposition method;Finally,extract the features of the fusion image for smile recognition.In order to verify the effectiveness of the algorithm in this chapter,the method proposed in this chapter is compared with the traditional single feature method and the method of literature[53-55].The experimental results show that the smile recognition rate of using the fusion LDP features is 96.69%,increased by 0.4%-6.69%;the recognition rate of using the fusion LBP features is 97.19%,increased by 0.9%-7.19%.Experiments show that the fusion method can improve the smile recognition rate and total recognition rate of smile recognition.3.Three kinds of facial expression recognitions of fusing visible light LBP feature and infrared LBP feature are proposed.Firstly,the visible light face image is divided into 6×5 regions,and the visible light LBP features of each region are extracted,then VLBP features are formed by them;the infrared face images were divided into six regions:forehead,left cheek,nose,right cheek,mouth and chin.The infrared LBP features of 6 regions were extracted and then ILBP features were formed by them;then the VLBP_ILBP features are fused with VLBP features and ILBP features by the end-to-end connection form.Finally,the recognition experiment of three kinds of facial expression is performed with using fusion features.The experimental results show that the recognition rate and the total recognition rate of VLBP ILBP fusion feature method increased among the positive expression and negative expression,compared with the VLBP ILPQ fusion feature,the VLPQ_ILBP fusion feature,the VLPQ ILPQ fusion feature and other traditional single feature methods,the method of fusion features proposed in this chapter can improve the recognition rate and robustness of expression recognition,to a certain extent.4.Three kinds of expression recognition based on the fusion of multi-feature and multi-classifier are proposed.Firstly,two sets of single feature classifiers are trained respectively by visible light LBP feature and the visible light LDP feature,and a set of fusion feature classifiers are trained by the VLBP_ILBP fusion features.When the classification results of two groups of single feature classifiers are the same,the classification results are outputted directly;if two classification results are different,then use the fusion features classifiers to reclassify,and the results of this classification are outputted as the final result.The algorithm in this paper uses the strong classification ability of the fusion feature classifier to solve the problem that the single feature classifier cannot classify correctly.Compared with the methods of traditional single feature and that of the previous chapter,the experimental results show that the algorithm of this chapter can improve the recognition rate of the three types of expression recognition,to a certain extent,and shows that the classification ability of multi-classifier fusion method is better than that of single Classifier classification.