Research On Students’ Emotion Recognition And State Analysis Based On Multimodal Fusion

With the rapid development of multimedia technology and artificial intelligence,the research and application of intelligent education technology based on deep learning and neural network are in full swing in recent years.In classroom teaching,learners’ learning mood and learning state are important indicators to measure learners’ learning effect and play a key role in students’ final learning quality.Students’ learning mood and state always exist in the whole learning process.However,due to the limitation of technology,it is difficult to observe students’ learning mood and state accurately and continuously in traditional teaching.Thanks to neural network technology,we can continuously and automatically identify and analyze the audio and video information of students’ learning.The main work of this paper is as follows.(1)Xception network is used to recognize expression emotion,convolution neural network based on time distribution is used to recognize speech emotion,and then multimodal emotion recognition prototype system is designed based on decision level fusion.Then,the first mock exam computer is used to collect the video of the students.The results of the video recognition are obtained through the system identification.The experimental results show that the accuracy of the emotion recognition results after fusion is higher than that of the single modal emotion recognition results.(2)After obtaining the results of multimodal emotion recognition,the pad value of the corresponding emotion was obtained through the pad score of students’ learning emotion.Then the pad Emotion Scale was used to establish the corresponding relationship between learning emotion and learning state,and the students’ learning state was judged according to the corresponding relationship.(3)A questionnaire survey was conducted to investigate the students’ learning status from the classroom learning and after class interest.On the one hand,in order to verify the effectiveness of the multimodal emotion recognition algorithm and obtain the real emotion of students in class;On the other hand,in order to investigate the factors that affect students’ class status,and improve teaching on this basis.(4)After six weeks of teaching application,on the basis of emotion recognition and questionnaire survey,this paper puts forward some teaching strategies for front-line teachers’ reference,so as to facilitate teachers to adjust teaching in time.The results show that the multi-modal information fusion emotional recognition system can facilitate teachers to check the students’ learning emotion and learning state in time.The real-time visual emotional analysis line chart can be used to observe the students’ emotional changes at any time,which greatly saves the analysis time and improves the timeliness and effectiveness of the emotional analysis.After six weeks of teaching application,students’ learning enthusiasm is gradually improved,and teachers’ classroom management is more convenient.