Engagement Recognition Based On Multi-Feature Fusion

Automatic engagement analysis is becoming an important cross-topic in the field of artificial intelligence and education.In intelligent interactive systems of artificial intelligence assisted education,humanoid service robot,robot partner/teacher,etc,it can be used to judge the learning state of students through the intelligent system,and provide feedback to teachers or parents,or serve as a decision basis for human-computer interaction.Compared with the traditional artificial recognition method or recognition method based on one single type of feature information,by using multiple-feature fusion combined with machine learning automatic engagement recognition method has obvious advantages.Automatic engagement recognition,however,is still a challenging problem.Its key lies in how to construct a reasonable framework and algorithm to extract and fuse many kinds of feature information to realize high-accuracy engagement recognition.This paper integrates face position,facial expression and eyes gaze information through effective multi-feature fusion strategy to construct a highly robust focus recognition model.The main research contents of this work are as follows:In terms of the definition of engagement,engagement matchs with observable visual cues such as posture and behavior to find a variety of effective information to reflect engagement.In this paper,facial expressions,eyes gaze and facial positions use as effective information to define different levels of engagement.In terms of making data sets,since there is no open source of engagement recognition data sets,making data sets is an important part.Collect the learning videos of 16 volunteers as the original data sources,and then process 62,792 images and 2425 video clips with 5seconds duration;Set screening rules to analyze and clean data;Set labeling principles to label the data as “In the focused state” and “In the non-focused state”.In terms of feature extraction,using a pre-trianed face detection model,based on Residual Neural Network(Resnet50),to extract the features of face location information and get all face images.Then feeding the facial images into a pre-trained facial expression recognition model based on Convolutional Neural Network(VGG-19),and feeding the raw images into a pre-trained eye detection model based on Stacked Hourglass Networks to extract the corresponding features of facial expressions and eyes gaze.In the aspect of feature fusion,a feature fusion strategy based on stacking model is constructed.Through the five-fold crossover stacking training,extract the effective features from different feature information by different classifiers,and generate more effective and representative fusion features,to improve the accuracy and robustness of the engagement recognition algorithm.In engagement classification aspect,using the engagement recognition framework to classify the engagement.Then calculate the confidence score of the category of “In the focused state”.Then divide the final level of engagement into “Best Engagement”,“Excellect Engagement”,“Good Engagement”,“General Engagement”,“Poor Engagment” five levels according confidence score.Test on the image data set,use the F1?score as the evaluation metrics,this work's F1?score of the engagement recognition based on fusion features is 93.47%,which is 1.03%higher than the optimal single type of feature that extracted in this work.The result shows that the fusion of multi-feature information is effective to improve the accuracy of attention recognition.Test on the video data set,use accuracy as the evaluation metrics,the accuracy of engagement recognition in this work is 92.8%,which is 16.48% higher than the current most advanced engagement recognition work based on single type of feature information and 17.5%higher than the current most advanced engagement recognition works based on multiple features.