Research On Feature Fusion Based Facial Expression Recognition Via Neural Network

Facial expression recognition(FER)can analyze emotional states of people and mental states of patients,provide expression feedbacks of users,and detect fatigue of drivers and work-ers,having broad development prospects in areas of medical services,interactive systems,sales of goods and services,security,etc.Although there are many research findings and progresses in FER,variabilities of intra-class expressions still severely restrict FER performance,especially among different individuals.The conventional single-feature-based models and feature fusion methods are difficult to solve this issue,but deep learning has become a more feasible tool for FER exploration since its rapid developments in the last few years.Based on the above background,this dissertation designs corresponding neural networks and feature fusion strategies to further extract and classify the facial expression features which are constructed with the help of facial geometry,wavelet transform and Fourier transform,ex-ploring new solutions for FER.The main research contents are as follows:1)A FER algorithm based on crucial region feature fusion is proposed.It firstly constructs a muscle movement model in order to better design and extract three normalized and repre-sentative crucial regions from facial image.Then,the Region-based Convolutional Fusion Network(RCFN)is designed to extract triple-level visual features from each crucial region and merge them for feature fusion and classification.This algorithm greatly suppresses variabilities of intra-class expressions by the crucial regions in the beginning of feature ex-traction,and strengthens the bond between low-level feature and semantic feature by the RCFN,having high accuracy and strong robustness.2)A FER algorithm based on fusion of geometry features and wavelet texture features is stud-ied.Firstly,fresh local-details geometry features are designed and extracted according to relationships between expression characteristics and facial landmarks in order to reduce intra-class difference and increase inter-class difference,and are further depicted by a light-weight neural network.Then,texture features from aforementioned crucial regions are con-structed by two-dimensional discrete wavelet transform,and are further extracted through improved version of RCFN.Finally,multiple fusion strategies are employed to explore the best recognition performance of these two kind of features.Comparing to the first one,this algorithm dose reduce the network parameters by over a half and effectively improve the accuracy and robustness.3)A FER algorithm based on facial geometry and frequency spectral features is explored.Firstly,both difference of local-details geometry features and difference of frequency data are extracted from the onset and peak frame of expression sequence respectively.Next,a light-weight neural network is applied to further depict difference of geometry features,while the newly designed broad convolution network,employed the central convolutional pattern,is used to extract frequency spectral features from difference of frequency data.Finally,these features are merged for classification.This algorithm suppresses variabilities of intra-class expressions by the differences extracted from dynamic information of ex-pression,while it stays efficient and light-weight with less than 850 thousand parameters,reducing 95% parameters comparing to the previous algorithm.4)A FER algorithm is conducted through the fusion of spatial texture and frequency spectral features.It first extracts the spatial texture feature via two-dimensional discrete wavelet transform and frequency data via Fast Fourier Transform from cropped facial image sepa-rately.Next,two pre-designed convolution networks are employed to further extract triple-level visual features from the above texture feature and triple-range frequency spectral fea-tures from the frequency data,respectively.Finally,these extracted features are combined for feature fusion and classification.This algorithm explores the complementarity of visual texture feature and frequency spectral feature,and solves the dilemma that the previous three algorithms rely on facial landmarks badly and only apply to frontal faces.The experimental results show the proposed four FER algorithms can effectively reduce or avoid inference from variabilities of intra-class expressions and reach current cutting-edge performance on the tested datasets.Among them,the second FER algorithm obtains the best performance comparing with other state-of-the-art methods.