Research On Multimodal Stress Recognition Based On Deep Learning

Stress will bring a series of negative effects to people,which are mainly manifested as tension,discomfort,anxiety and depression in psychology and medicine.With the acceleration of the pace of life,the physical and psychological problems caused by stress have become more and more significant,which has seriously threatened people’s health.Therefore,it is of great significance to carry out research on stress recognition.In recent years,the research of multimodal stress recognition has made some progress,but the following problems still exist:(1)The data set of multimodal emotional stress recognition is not rich enough;(2)The multimodal feature representation is relatively simple;(3)Insufficient fusion of multimodal information;(4)Modal data may be missing and affect model performance.Aiming at the above problems,this paper selects two modalities of facial expression and pulse signal for stress recognition research.The main work is as follows:(1)In this paper,a multimodal stress data set is established.The stress inducing experiment is designed using the psychological experimental paradigm,and the facial expressions and pulse signal data of 60 college students are collected.(2)This paper proposes a multimodal stress recognition method based on multi-scale attention and tensor fusion.Most of the features of existing pressure recognition algorithms are manually extracted,and feature fusion is not sufficient.In this paper,convolutional neural networks are used to extract the features of multimodal data,combined with multi-scale attention mechanism to enrich feature expression,and highlight information related to pressure.A tensor fusion technology based on low-rank decomposition is introduced to express and fuse the vector features of different modalities to form high-order tensor features,learn the information within and between modalities,and use low-rank decomposition to accelerate the fusion process.The experimental results show that the stress recognition method proposed in this paper shows good performance,and the recognition accuracy rate reaches 87.68%.(3)This paper proposes a multimodal stress recognition method based on time attention.pressure is a dynamic process in time.This paper proposes a Bi-GRU-based time attention mechanism to extract the time series characteristics of data,and calculate the attention distribution of each modal information at each moment in real time to capture the pressure Changes in time.The experimental results show that after adding the time attention mechanism,the model’s pressure recognition accuracy rate reaches 88.83%.(4)This paper proposes a multimodal stress recognition method based on privilege learning.In order to solve the problem of the lack of modalities caused by the difficulty of obtaining pulse data in actual situations,this paper presents a multi-modal privilege learning algorithm that uses a multimodal(Facial expression + Pulse signal)model to assist singlemodality(Facial expression)model training.The multi-modal model can still work normally when it degenerates to a single-modal model due to missing pulse signal data.The experimental results show that after using the multimodal privilege learning algorithm,the accuracy of model recognition is improved by 1.5% compared with the model trained with only facial expression data.In this paper,a multimodal stress data set has been established,and certain explorations have been made in the aspects of multimodal representation,fusion,and collaborative learning,which is conducive to the application of stress recognition technology in real life.