| Emotion recognition aims to identify human emotions using information from various channels.As a research hotspot in affective computing,automatic emotion recognition has shown potential applications in many fields,such as smiling face capture,fatigue detection.assisted psychological counseling and treatment,and intelligent advertisement push.Early researches on emotion recognition,especially facial expression recognition,were mostly based on single-label classification research on datasets of seven basic emotions collected in laboratory environment.However,there is a large gap between the controllable collection environment and practical application scenarios.Emotion recognition research for real-world applications has gradually transitioned from a single controllable environment to complex and uncontrollable natural scenes.In the real world,people do not exist independently,and human emotional expression is inevitably affected by the surrounding environment and events.Although,emotional information can be obtained from human emotional expressions such as facial expressions,body posture changes and voices,the camouflage of human emotions and the problems of illumination and occlusion in the real environment also bring great challenges to emotion recognition research.Facing emotion recognition in natural environment,this paper firstly studies facial expression recognition and context emotion recognition using surrounding environment information based on static images.The process of emotional change is often continuous and time-correlated,and the ability of sequence expression to capture emotion is more comprehensive and specific than a single static picture.Therefore,this paper creates a context-based video emotion dataset,expands the basic emotion categories,and performs fine-grained emotion annotation.Then,the proposed method based on static images is extended to the dynamic emotion recognition task,and a multi-modal and multi-branch emotion recognition system is constructed and evaluated on our dataset.The main work of this paper is as follows:(1)To address the limited flexibility of facial pre-defined regions of interest and the large scale of the backbone network model used in most methods.A lightweight network based on difference saliency maps was proposed.In this method,difference saliency maps are generated by using the difference image between neutral face image and action face image.Therefore,the predefined regions of interest do not depend on facial landmarks,and can focus on multiple face regions at the same time.The lightweight network is based on group convolution and skip connections,which not only reduces the size of the network but also ensures the recognition effect.During training and prediction,the difference saliency maps are added to the neural network layer by layer as pixel-level guidance information to enhance the effective information of feature maps.Experiment results on three different datasets including Aff-wild2 show that the proposed difference saliency maps can effectively provide a priori guidance information for facial action unit detection.The lightweight network is effective in facial expression recognition and action unit detection.At the same time,further experiments demonstrate that our network is more efficient in using parameters,computation complexity and inference time.(2)To address the problem that only visual information is used in current contextual emotion recognition methods,by incorporating external structured emotion commonsense knowledge,two methods are proposed for constructing emotion knowledge graphs based on the objective text of images,and a multi-modal emotion recognition model is designed.By converting visual information into objective text descriptions,the subject and object relationship information useful for emotion recognition are retained,and a large amount of irrelevant and redundant information is filtered to reduce the burden of the model.The obtained objective text is fused with the structured emotional commonsense library SenticNet6.A large-scale emotional knowledge graph is obtained based on data-driven methods.A small-scale textual emotion knowledge graph is obtained based on slidingwindow-based method.A multimodal emotion recognition model is constructed using two designed emotion knowledge graphs and the subject’s pose keypoint information as input.In addition to the subject,the pose information of the two persons with the greatest intimacy with the subject is also used.Therefore,five modalities are used in multimodal emotion recognition model when social relations are considered.In addition,two fusion modules are proposed,one is attention-based and the other is a deep reasoning module that includes interpersonal relationships.Experiment results show that the emotional knowledge graph integrated with structured commonsense knowledge contains important image contextual scene information.The multimodal emotion recognition network is superior to the most advanced methods.And the relationship between interpersonal relationships can also be modeled to improve emotion recognition.(3)In view of the small scale of video emotion datasets,few emotion categories,and the lack of consideration of the impact of events in a wide time range on emotion,this paper constructe two video emotion datasets.Among them,the HEU Emotion dataset is collected from video sources such as online social media and film and television resources,and a largescale natural scene video emotion dataset containing ten emotions is obtained through data screening and annotation.The SVCEmotion dataset is manually cropped and annotated from film and television resources by data collectors and annotators.The characters in the videos are tracked through an automatic pedestrian tracking algorithm,and the specific location of each annotated person is given.In addition,SVCEmotion also contains a variety of annotation information,such as subtitles,situation descriptions,fact descriptions,scenes,events,personal attributes and relationships of characters,emotion categories,emotion polarity,and activation values of six basic emotions.Finally,a fine-grained multi-label video emotion dataset containing 28 types of emotions is constructed,which expands the emotion expression range of the classical discrete emotion description model.Situation descriptions,personal attributes and social relationships in the annotation information provide a data basis to personalize emotion perception and understand emotion in a wide range of time for emotion recognition.(4)Using the information of various modalities in the SVCEmotion dataset,visual,audio and text,a multi-modal emotion recognition system is constructed.To address the uncertainty caused by the image quality in the dynamic expression recognition problem,a feature selection fusion module is proposed.In this paper,the facial expression recognition model and posture emotion recognition model are first trained on the large-scale video emotion dataset HEU Emotion to obtain pre-trained parameters,and then transferred prtrained parameters to the SVCEmotion dataset.And I3D.VGGish and ABN models and global emotion labels of videos are trained.In addition,the emotional knowledge graph construction method in Chapter 3 is used to construct emotion graphs for the situation description and fact description in the SVCEmotion dataset.Finally,the information of multiple modalities is fused.Experiment results show that pre-training on large-scale emotion datasets is very necessary.The proposed feature selection fusion module does not introduce too many parameters but can effectively fuse sequence information.The experimental results based on contextual description show that information in a wider time range can provide effective prior knowledge for emotion recognition in videos,thereby improving the recognition effect. |
