Expression Recognition Through Implicit Integrating Multi-spectral Facial Images

Recent years have seen increased attention on facial expression recognition due to its wide application in many areas of human-computer interaction. Most facial expres-sion recognition research focused on visible spectrum, which is sensitive to illumination changes. While thermal images, recording facial temperature distribution, are robust to light conditions. Therefore, expression recognition by visible and thermal image fusion is promising. However, in most cases, only visible images are available, since thermal cameras are much more expensive than visible cameras, which are popular in our daily life. Thus, in this paper, we propose novel visible expression recognition approaches by using thermal infrared data as privileged information, which is only available during training. In addition, this thesis makes a preliminary discussion on the generation of thermal infrared images. The details are as follows:(1) We propose a novel visible expression recognition approach, expression recog-nition through thermal-augmented visible features. First, active appearance model pa-rameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from thermal infrared im-ages. Second, feature selection is performed using the F-test statistic. Third, a new visible feature space is constructed using canonical correlation analysis under the help of thermal infrared images. After that, a support vector machine is adopted as the clas-sifier on the constructed visible feature space. Experiments on the NVIE and Equinox database show the effectiveness of the proposed methods, and demonstrate that thermal infrared images' supplementary role for visible facial expression recognition.(2) We proposes a novel expression recognition method, recognition from thermal-augmented expression classifier. Thermal infrared images available during training are exploited to construct a better facial expression classifier from visible images. Specif-ically, visible expression classifier and thermal expression classifier are learned simul-taneously during training by adding similarity constraint on two expression classifiers.The support vector machine is modified by the similarity constraint to improve visible expression recognition. Efficient learning algorithm of the proposed method is also de-veloped. Experimental results on the NVIE and Equinox databases demonstrate that the proposed thermal augmented expression recognition method can effectively exploit thermal infrared images' supplementary role for visible facial expression recognition during training to construct a better visible expression classifier, and thus outperforms existing works.(3) We propose a method for generating infrared images based on a visible images,and an expression recognition is performed based on the generated image. The used framework is Generative Adversarial Network. The U-net and the encoder-decoder structure are used as the structure of the generator, respectively. The convolution neu-ral network is used as the discriminator. Two object generation functions are used to generate the images and the results were compared. Experimental results on the NVIE datasets show that the framework, Generative Adversarial Network, can be used to gen-erate infrared images from visible images. Pixel-level constraints can improve the true extent of the generated image.