Research On Dual Modal Learning Situation Recognition In Resnet And DS Evidence Fusion

In the information age,computers which achieve effective human-computer emotional interaction are expected to perceive each other's emotions as intelligently as humans.Traditionally,human emotions which are analyzed through expressions,sounds and other single modal information often has certain limitations and inaccuracies,for human emotions are often the comprehensive presentation of multiple modal information.In recent years,the use of multimodal analysis techniques in emotion recognition has become a hotspot in the development of artificial intelligence.In the field of education,grasping the learning situation of students accurately in class is conducive to enhancing the effect of classroom teaching.In the classroom,the emotion expressions of students are often the synthesis of multiple modal information,so it is particularly important to analyze the multi-modal information.In this paper,two modal information of their facial expressions and sitting postures in the classroom are selected as the basic indicators for evaluating students' classroom emotions.In order to grasp their emotions in real-time,an improved Resnet is designed,the DS evidence theory is used to analyze the recognition of the fusion of the two modal information.The confusion matrix acts as a bridge,linking the improved Resnet and DS evidence theory.The main work done in this article is:1.An improved network with residual modules?Resnet₅0_v2?is designed.The traditional convolutional neural network used for recognition and classification will show gradient disappear or gradient explode with the increase of the number of network layers,so that the cost function cannot converge as expected.For the sake of avoiding this situation,the basic idea of Resnet is adopted in this paper.When the function H?x?is approximate to the identity transformation,it is better to fit the residual function F?x?=H?x?-x,it is easier to train and has been improved on this basis."Pre-activation" method is adopted as an improvement method to make the information in the back propagation process unimpeded,and enhances the generalization ability of the model.2.Data enhancement of this data set is realized in two ways.The original image data used in this paper is collected by mobile phone.Considering the model's need for data scale,the data set is enhanced by two methods.The first method is based on digital image processing;the second method is based on generating adversarial network.Because the two data enhancement methods allow the data to be expanded,the model has better generalization performance.3.The improved DS evidence theory is used to achieve two-mode information fusion.The confusion matrix identified by the single modal information of the students'facial expressions and sitting postures in the classroom is normalized as the initial probability distribution function;and two modal information of facial expressions and sitting postures are used as two basic evidences of the DS evidence theory fusion analysis of dual-modal emotion recognition.In view of the shortcomings of the classic DS evidence theory,the improved DS evidence theory is introduced to make the distribution of evidence more reasonable and the fusion result more convincing.The confusion matrix is normalized and the weight of evidence is redistributed through the improved DS evidence theory.The Pignistic probability transformation method is used at the decision fusion layer to make the final decision.4.The experimental results are analyzed.Facial expressions and sitting postures are used as the basic input of the network constructed in the early stage,and through network training,the recognition result of single modal information is obtained.Then with the confusion matrix of Resnet is processed and combined with DS evidence theory for experimental analysis,the accuracy of the experimental results is 81%.Compared with other algorithms,the performance of the algorithm in this paper is more better.