Emotion Recognition Research Based On Hyperspectral Image

Emotion plays an extremely important role in human perception,decision-making and many other processes.More and more scholars begin to pay attention to the research on human emotion.The rapid development of information technologies provides endless opportunities and challenges for affective computing,i.e.the development of wearable technology produces more and more feasibility for recognizing and modulating emotional states.Extensive researches and applications have been carried out in medical care,remote education,intelligent robots,intelligent communities and other fields;therefore,emotion recognition has received great attentions from academia and industry.In emotion recognition studies,facial expression and physiological signal are considered as the most important modalities.The researches on emotion recognition of single modality(facial expression or physiological signal)have been widely applied,but there are still many disadvantages for themselves.For example,facial expression is well visualized and feature information hidden inside facial expression is rather rich,but facial expression can easily cover up the real inner human experience and its continuous capture is also difficult in reality;emotional measurement of physiological signal is more real and reliable,and it is not easily controlled by subjective consciousness,but its measurement conditions are considerably complex and changeable,and noise redundancy tends to be large.A large number of studies have shown that the complementarity,uniqueness and relevance of modal fusion can improve the system stability and accuracy.These characteristics prompt researchers to consider modal fusion for emotion recognition.The thesis proposes to use the effective fusion of facial expression and facial Tissue Oxygen Saturation(St O₂,a typical physiolocial measurement)for emotion recognition,aiming at developing the recognition performance.The thesis starts from the understanding of using spectral imaging to recognize emotions,and then performs in-depth emotion recognition researches from three aspects:facial expression,physiological signal(St O₂)and their modal fusion.During these research procedures,some detailed knowledge explanations have been introduced:the construction process of a new texture feature operator;the calculation process of St O₂;the establishment of St O₂ and expression databases.Based on the above-mentioned researches,a non-invasive emotion recognition that combines facial expression and St O₂ is finally proposed.Specifically,this approach first applies Hyperspectral Imaging(HSI)technique to contactlessly capture structural information of facial region,and then extracts their corresponding features of facial expression and St O₂ from HSI image cubes,respectively.Based on Support Vector Machine(SVM),Collaborative Representation Classifier(CRC)and Multilayer Perception(MLP),an integrated classifier model is designed to learn the recognition process.Finally,a modal-fusion-based emotion recognition system is then constructed.The main research contents of the thesis are present as follows:1)This paper applies HSI technique to perform non-invasive emotion recognition,overcoming the problems of insufficient comfort,poor concealment and inconvenient operation brought by traditional contact device during data collection,which could affect the result of emotion recognition.HSI emotional database is constructed using HSI technique,and the specific work include:experimental design,platform building,data collection,data processing and transformation,and database establishment of emotional St O₂ and facial expression.Based on these work,a modal-fusion-based non-invasive emotion recognition research has been investigated in detail.2)This paper proposes a new pattern operator(Histogram of Local Dynamic Texture Patterns,Ho LDTP),taking global and local texture feature patterns into consideration,and subsequently describes its feature extraction process in detail.The operator is applied to feature extraction,and its effectiveness is experimentally proved through face recognition and facial expression recognition.The experimental results show that Ho LDTP can better represent the spatial local texture feature information of the image,and has a good performance for pattern recognition.Therefore,it is a well promising pattern operator.3)This paper proposes to use St O₂ to carry out the research on emotion recognition.Considering a large number of relevant studies,facial region is firstly re-segmented to obtain 19 new facial Regions of Interest(ROIs)segments for emotion classification.On this basis,facial features are extracted using this segmentation method(i.e.St O₂ feature vector is constructed),and then the experimental results are analyzed.The results prove that emotional changes cause the regular changes of St O₂,and it is effective and feasible to use St O₂ to recognize emotional states.In addition,a new facial ROI segmentation method proposed in this paper provides a directive reference for future ROI research,and is conducive to the ROI based pattern recognition research.4)This paper proposes a modal-fusion-based non-invasive emotion recognition method that integrates physiological signal(facial St O₂)and facial expression.The image cube collected by HSI can not only reflect the texture pattern but also the physiological pattern.Combining their advantages(St O₂ can objectively reflect emotional changes;texture pattern inside image is rich.It can complementarily intergrate their respective advantages),the modal-fusion-based emotional learning is carried out,and the distribution patterns of emotional states can be mined from different perspectives.HSI data has potential application value for emotion recognition,and its rich information can effectively enhance the predictive ability of the recognition system,which has positive supporting role for constructing a feasible practical model in the future.The research proves that the modal fusion of facial expression and physiological signal(facial St O₂)has a higher accuracy in the classification of four emotional states(anger,calm,happiness,and sadness),and can improve the consistency of emotion recognition.This fusion method makes reasonable use of the emotional information in HSI data.Compared with the recognition results achieved only on the single modality(facial Ho LDTP or St O₂),the modal fusion can develop the classification accuracy.Moreover,the modal fusion can provide more stable and reliable recognition performance with respect to the single modality.The research shows that the modal fusion is more effective for emotion recognition and has important theoretical value and potential application prospect.