Multimodal Depression Recognition System Based On Expression And Text

In recent years,depression has become a worldwide common mental illness.At the same time,there are many difficulties in the clinical diagnosis of depression-related mental diseases.There are more and more researches on the application of artificial intelligence algorithms to help identify depression,Methods based on facial expression,text,speech,eeg and other data modes also emerge in an endless flow.It is feasible to use AI technology to assist the diagnosis of depression by fusing data of various modes.(1)In this thesis,artificial intelligence technology is applied to the research of assisted diagnosis of depression,and a multimodal depression recognition system based on facial expression and text is implemented.The main work of this thesis is as follows:(1)Building a data set containing facial expressions and text semantic information.By collecting videos of patients’ real consultation process in hospitals and designing fast and reliable labeling methods,an original semantic data set containing more than 30,000 pictures with labels of depression was built.(2)Based on FACS facial action coding system and CNN-LSTM network,a new CLAD facial expression recognition model for depression was proposed,and the text emotion classification based on BERT pre-training model was implemented.The text content was converted into word vector through the pre-training model,and then the text word vector was weighted by feature.After constructing the sentence vector,it classifies and recognizes it.(3)Two single modal identification module results in policy makers weighted fusion inputs to the classifier,the resulting blend mode classification of depression as a result,to realize the whole multimodal recognition model of depression.Compared with the clinical diagnosis results of 228 depressed subjects and 109 non-depressed controls,the model achieved 75.9%classification accuracy,which verified the effectiveness of the method.