Multimodal Physiological Data Analysis For Emotion Recognition

Emotions direct our day-to-day activities.Affective characteristics can aid in driver status monitoring,mental health diagnoses and defense systems.The main aim of this study is to investigate multiple physiological signals affective characteristics collected from subjects as well as using them to recognize perceived emotions in a valence-arousal dimension.This dissertation focuses on emotion recognition based on physiological signals and the main contributions are as follows:Firstly,an important key to accurate emotion classification is the reliability of extracted features.We propose a hyper enhanced feature learning system(HELS)to power physiologically extracted traditional features from multi-modalities to form a hybrid neuromultimodal network.Instead of taking data directly,our models take features as mapped input to reduce structure complexity and save memory.The model is also capable of randomly and automatically generating and updating weights with enhancement nodes to present more informative feature nodes for emotion classification.Experimental results with artificial neural network as classifier show the superiority of our work with existing ones.Secondly,we propose an effective gender-age relation graph(GARG)model based on EDA signals.The graph is constructed using subjects’ observed gender and age information as embeddings to capture relations between given entities.We also extracted statistical features(SF)consistent with cognitive studies.We then combined the statistical features,and the knowledge graph features(SF-GARG)in attempt to classify emotional states.The model uses deep learning techniques to classify emotional responses of subjects.In addition,we propose a feature fusion technique that exploits the knowledge embedding vectors as neural weights to statistical features.When compared to other methods,our proposed method shows more robustness and better emotion recognition accuracy.Finally,this dissertation investigates the effects of auditory and visual stimulation commonly used in studying emotional responses of subjects to understand brain regions where emotions are mostly stimulated.We propose a brain region aware domain adaptation(BRADA)algorithm to tackle subject-to-subject variation and mitigate distribution mismatch by normalization and augmentation across databases.BRADA algorithms treat features from visual and auditory channels differently and works with existing transfer learning methods.The model eventually trains one database and tests on a different but related new database using knowledge acquired from the first data to improve model performance and vice versa.The proposed model clearly reduces constrains on existing affective brain computer interfaces(aBCI)and is practically feasible in real life applications.