HDC-ASER:Automatic Student Engagement Recognition Based On Hybrid-dimension Convolutional Neural Network

The development of the Internet has profoundly changed all aspects of our life,including clothing,food,housing and transportation.More importantly,it has changed our way of learning.Our learning environment is not limited to books,classrooms and teachers.We can study both in the classroom and online.However,the drop-out rate of online learning platform is very high,and the teaching effect of traditional large classroom is not small class teaching or one-to-one teaching.The reason is that in these learning environments,the good communication between teachers and students is cut off,students'participation and concentration in learning can't be fed back to teachers in time,teachers can't receive students' learning state response,so they can't make timely intervention and flexible classroom adjustment.Therefore,we find that it is of great significance to identify classroom concentration for classroom teaching or online course learning.The degree of students' concentration reflects the degree of students' concentration in class,their acceptance of and interest in the knowledge they have learned.He is closely related to students' academic performance,which is of great significance to reduce the dropout rate of online teaching platform and personalized online courses.The existing methods of student engagement recognition include methods based on questionnaire,sensor,click stream and video data.However,methods based on questionnaire and sensor are simple and intuitive,but they can not be extended to large-scale online classroom.Methods based on click stream data focuses on whether students drop out of online courses or finish their courses.It ignores the intermediate courses of learning and cannot be applied to the traditional teaching environment.In recent years,methods based on student learning video is a new trend in the field of student focus recognition.The method of recognition of students' concentration based on video analyzes students' learning video by means of machine learning and deep learning.The existing methods of student focus recognition based on video are mainly divided into two-dimensional convolution method and three-dimensional convolution method.Although two-dimensional convolution has achieved the peak effect in image classification,its lack of natural structure makes it difficult to model and analyze the three-dimensional video data sequence.Although 3D convolution can model the time and space information of students' focus video at the same time depending on its structural advantages,the volume of 3D convolution becomes extremely large.Compared with 2D convolution with the same layer,3D convolution has more parameters for training.Based on this background,this paper discusses a method of students' concentration recognition based on mixed convolution mode.The so-called mixed convolution is to use two-dimensional convolution and three-dimensional convolution in the same convolution.The hybrid convolution overcomes the weakness of the pure two-dimensional convolution model which is difficult to model the time series,and also avoids the disadvantage of the large number of parameters of the pure three-dimensional convolution model which is difficult to train.Combined with the problems existing in the field of students' engagement recognition,such as small data set,unbalanced data,weak optical flow information,short duration of concentration,and large changes in video lighting,this paper proposes an automatic students' engagement recognition methods based on hybrid-dimention convolution(HDC-ASER).The main work is as follows:1.Based on the mixed convolution network,the model of automatic recognition of students' concentration is proposed,which overcomes the shortcomings of the pure two-dimensional convolution network and the three-dimensional convolution network.And in the data set of the field of students' focus recognition,the score is higher than the benchmark.The results of four classifications were 60.2%and two classifications 97.6%.2.Considering the weak information of data optical flow in the focus area and the large amount of data redundancy between frames,this paper designs a reasonable frame sampling frequency and three-dimensional convolution kernel size.The best sampling frequency and the most suitable mixed convolution structure are obtained.3.Considering the serious data imbalance in the field of students' concentration,this paper extends the focal loss for the two-classification data imbalance to multi classification,introduces the focal loss of multi classification,and solves the serious data imbalance in the field of students' concentration.4.For students' concentration video,students' learning background environment changes greatly,such as large span of lighting conditions,many and miscellaneous backgrounds.This paper overcomes these problems by means of data enhancement.The preprocessed data can be directly put into the model for learning.5.In order to prove the expansibility and practicability of the model proposed in this paper.In this paper,HDC-ASER model is also trained and validated on the AFEW dataset,and the result is higher than the benchmark.