Facial Expression Recognition With RGB-D Image Based On MTMW-SOS-ELM

With the rapid development of machine learning,artificial intelligence technologies are widely used in our daily life,which greatly improved the intelligence of living environment.Using machine learning algorithms to recognize facial expression is a researching focus,which improves human-emotion awareness ability of computers.Nowadays,standardized laboratory-captured facial expression images can be recognized very wel.But for natural environments,people show various facial actions for a same expression,so recognizing natural expression is much difficult than standardized expression under the influence of subject-specific facial action differences.Thus,natural expression recognition is a difficult problem,it needs lots of improvement for expression recognition algorithms.This paper used a Kinect 3D-Image capturing device to capture facial RGB-D images of experimental subjects,with the consideration of the assistant roles of depth images.To ensure the captured datasets are closer to natural environment expression,this paper do not require experimental subjects make standardized facial actions for each expressions,but let them make facial actions naturally by themselves.For avoiding the influence of subject-specific facial action differences,improving non-constraint expression recognition ability,and accelerating realtime RGB-D fusion expression recognition at the same time,this paper referred a former work SOS-ELM?Semi-supervised Online Sequential Extreme Learning Machine?algorithm,added multitask and multiway method to SOS-ELM,and finally proposed MTMW-SOS-ELM?Multitask Multiway Semi-supervised Online Sequential Extreme Learning Machine?algorithm to implement realtime natural expression recognition for personal computer users.Main researching works and ideas in this paper are:1^st.Feature extraction on depth images.This paper implemented fusion expression recognition which uses texture feature of both RGB image and depth image,based on facial RGB-D image captured by Kinect.This paper referred Haar-like texture feature extraction algorithm,and extracted the feature of RGB-D images on red,green,blue and depth sub-image individually.The experimental result shows,to be compared with using RGB images only,utilizing texture features on depth images will increase the separability of the expression feature,it improves the expression feature extraction ability.2^nd.Proposed a MW-ELM?Multiway Extreme Learning Machine?neural network structure,which accelerates facial expression recognition,to meet the requirement of realtime RGB-D facial expression recognition.For RGB-D fusion expression recognition,this paper proposed MW-ELM model,it separates input layer to 4 ways,and input the Haar-like feature extracted on red,green,blue and depth sub-image separately to each way.With the consideration of relative independence of each sub-image,for avoiding the massive computation of fully-connected neural network,this paper separates hidden layer to 4 ways too.Hidden nodes of each way only processes feature vector of its corresponding sub-image,and multiway fusion is implemented on output layer.According to theoretical analysis and experiments,after this improvement,with basically same recognition accuracy,MW-ELM uses only 1/4 recognition time than ELM?Extreme Learning Machine?.MW-ELM greatly improved the expression recognition speed.3^rd.Proposed a MT-ELM?Multitask Extreme Learning Machine?neural network structure,to overcome the impact of subject-specific facial action differences,and improve generalization ability of the facial expression recognition model.With this multitask training method implemented on ELM,the impact of subject-specific facial action differences will be reduced.It will overcome the training subject dependency of ELM,improve the learning ability of subject-independent common expression feature,and it can recognize expressions for untrained subjects.4^th.Implemented a MTMW-SOS-ELM?Multitask Multiway Semi-supervised Online Sequential Extreme Learning Machine?algorithm,and applied it to realtime RGB-D natural expression recognition.At first,this paper referred a former work SOS-ELM and implemented a facial expression recognition system that supports semi-supervised learning and online sequential learning.Then,based on it,this paper combines multiway and multitask neural network structure and training method with SOS-ELM,and implemented MTMW-SOS-ELM algorithm for realtime natural expression recognition.Beside this,this paper experimented on a self-captured RGB-D natural expression dataset.For trained subjects,the average recognition accuracy of their test data is 91.73%.For untrained subjects,the average recognition accuracy is86.54%.