Facial Features And Body Movements Based Multi-modal For Depression Recognition

Depression is a common and high-risk mental disorder,and the existing depression assessment methods are complex and relatively subjective,so an effective diagnosis way is urgent needed.Facial activity is the main external manifestation of people’s emotional changes,and facial features can be used to study the psychological activities and mental state of patients with depression.Patients with depression are affected by the disease for a long time,which show slow movement,decreased body coordination ability,delayed response and so on in the clinical manifestations of body movements.This paper is based on facial features,body movement and multi-modality for depression recognition research.The main work is as follows:(1)Depression recognition based on facial features.Facial feature data is collected in the scoring process of Hamilton Depression Scale.In this paper,video frame extraction method is used to process the videos,so as to align the data dimensions;Openface tool is used to extract the facial feature data and normalize the data preprocessing;through the analysis of facial feature data,the differences of facial features between the depression group and the control group are found;and the proposed model CNN-LSTM for facial feature extraction based depression detection is constructed which gets the 75.56% accuracy.(2)Depression recognition based on body movement data.Body movement data is collected by Kinect device in the process of body movement stimulation task.After extracting human skeleton data,noise elimination and data smoothing are carried out.Through the analysis of body movement data,a method for depression detection based on simple movement task is proposed.By adjusting the size of dilated convolution scale and residual block design,this paper constructs a depression recognition model based on temporal convolution network(TCN),which is good at learning temporal and spatial features of body movement data,and the accuracy is 71.11%.(3)Multi-modal depression recognition based on feature fusion.In this paper,through the facial information feature vector extracted by CNN-LSTM and the body action feature vector extracted by TCN,the multi-modal feature fusion method is used to realize the result prediction and achieve the effect of depression recognition at the decision-making level;the accuracy rate of multi-modal depression recognition model reaches 82.22%.By comparing the results of single-mode and multi-modal depression recognition,it is found that multi-modal has more advantages in depression recognition.In this paper,a multi-modal depression recognition model based on facial features and body movements is established,which is expected to be suitable for psychiatrist’s auxiliary diagnosis of depression.