Virtual Reality Emotion Recognition Based On Weight Of Panoramic Multi-view Regions And Audio-visual Fusion

With the development of virtual reality(VR)technology,the application of VR content in education,medical treatment,multimedia digital entertainment and other fields has been widely concerned.It is important to research the emotional perception of users under the VR environment to optimize content and improve user experience.Therefore,this paper focuses on the problem of emotion induction and emotion recognition in VR scene.At present,there are several obvious problems in the research of emotion induction and emotion recognition in VR scene.Firstly,traditional emotion inducing materials are mostly in image,audio,video and other formats.There is no scientific method to establish the mapping between VR scene and emotional label.Secondly,compared with traditional multimedia materials,VR scene has a great difference in visual interaction.Therefore,it is urgent to study the representation of visual features and explore the relationship between features and emotion.Thirdly,traditional research focuses on image,audio,video and so on,and there are very few frameworks of emotion recognition model suitable for VR scene in the field of emotion recognition.This paper carries out the following three aspects for the above problems:(1)Establish the VR emotion inducing material library and its data set containing images and audio.This paper sets up a framework of material library based on multiple interactions,selects appropriate emotional features from standardized emotion inducing materials,and guides the design of emotion inducing materials in VR.Through series of psychological experiments,the standard and effectiveness of materials are proved.Finally,a VR multi-modal data set is established.(2)According to the distribution of visual space in virtual reality scene,this paper proposes a method of panorama emotion recognition based on the weight of view regions.The panorama’s space is divided into six view regions and weighted,which is used to help fine tuning convolutional neural network to extract emotional features.Finally,the classification algorithm is implemented to get the emotion recognition results.The accuracy of emotion recognition reaches 95.32%,which proves the validity of the method.(3)Aiming at the VR scene with low resolution or low frame rate,the multi-modal fusion mechanism is introduced according to the characteristics of visual-auditory dual channel input in VR scene.This paper establishes a VR emotion recognition method based on visual and auditory fusion.The method extracts the emotional features of visual and auditory content,and then fuses the auditory features and the visual features in public space.Finally,the emotion recognition results are obtained by classification network.Through experiments,the accuracy of VR emotion recognition method reaches 97.15%,which is better than that of single mode method.Through the above aspects,this paper systematically studies the emotion recognition of VR based on weight of view regions and visual-audio fusion.Besides,the material library and algorithm provide a reference for future research.