Teacher Emotion Recognition Based On Multimodality In Online Teaching Environment

With the development of intelligent technology,online teaching has gradually become an important application scenario in the field of education.Research on teacher emotion in online teaching environments is urgently needed.Researchers hope to use data from online teaching scenarios to automatically recognize teacher emotions through machines,in order to help teachers achieve precise teaching and improve teaching effectiveness.Teachers can also use this data for post-class reflection and as an indicator to evaluate their own teaching level.By accurately portraying the emotions of online teaching teachers,we can better understand the actual situation in the classroom and improve objective evaluations of the classroom.Online teaching is a rising field in recent years,with little research on the emotions of online teachers.Existing classroom emotion recognition studies mostly use recorded videos of the entire class to study multiple aspects such as speech,text,facial expressions,and body movements.Considering the uniqueness of online teaching classrooms,this study will focus on facial expressions and speech as two modalities for teacher emotion recognition research.By obtaining images and voice data of teachers during online teaching videos and using a multimodal fusion method to achieve multimodal emotion recognition of images and speech,we aim to accurately recognize the emotions of teachers during online teaching.To achieve this research goal,this paper mainly undertakes the following work:1.For the task of multimodal emotion recognition in online teaching,a dataset of online teaching teachers was designed.First,a batch of online teaching videos was selected on video websites,including various types of teaching scenes and topics.Then,these videos were trimmed to remove irrelevant content,ensuring the purity and effectiveness of the video samples.Finally,five evaluators who passed the credibility test were invited to annotate the emotional content of these video samples,resulting in a basic dataset of 490 video samples.2.A K-Bayes algorithm was proposed to classify the HOG features extracted from processed videos.The classification results of this algorithm were compared with those of the KNN and Naive Bayes algorithms.Additionally,a classic decision fusion algorithm was used for comparison with the K-Bayes algorithm.The results showed that the K-Bayes algorithm demonstrated good recognition performance.3.A method based on multi-class support vector machines(SVM)was used to classify emotional states based on the extracted prosodic features from speech.Two SVM methods were employed,namely,multi-class SVM with multiple binary classifiers and one-versus-one SVM,and their classification performance was compared.Experimental results indicated that the multi-class SVM with multiple binary classifiers method outperformed the one-versus-one SVM method in emotional state classification.4.A decision-level fusion strategy for multimodal fusion was explored,and the shortcomings of classical decision fusion and traditional D-S evidence theory were described.To address this,an improved D-S evidence theory was proposed and used for experimenting with multimodal fusion.The experimental results demonstrated that the fusion results of the improved D-S evidence theory were superior to those of the single-modal recognition model and exhibited good recognition performance.Furthermore,a comparison was made between the improved D-S evidence theory,traditional D-S evidence theory,and classical decision fusion,and the results showed that the improved D-S evidence theory achieved better recognition performance.