Research On Mental Fatigue Detection System Based On Multi-Source Domains Transfer Learning

Working on cognitively demanding tasks,such as driving and surveillance through video,for a considerable time often leads to mental fatigue,which leads to the decline of operators’ cognitive ability and performance like reaction speed.Such mental fatigue has a negative impact on work efficiency and quality,and even becomes a key factor in safety accidents.Therefore,it is of great significance to build the real-time detection system for mental fatigue under high cognitive load.Previous studies have found that the detection of mental fatigue can be achieved based on electroencephalography(EEG),electrocardiogram(ECG)or video data.However,due to the large inter-subject variability of mental fatigue-related features,the detection ability of models always reduce greatly in the face of unknown subjects,and cannot be widely used in practice.Therefore,improve the cross-subject detection ability of the model becomes the key to the problem.In this paper,we propose a mental fatigue detection method based on multi-source domains transfer learning with multimodal(EEG,ECG,video)input.Improve the cross-subject detection performance of the model by domain adaptive transfer learning method.The model performance verification results show that the cross-subject mental fatigue detection performance of the proposed multi-source domains transfer learning model improved about21.62% compared with the non-transfer multimodal model.Compared with the traditional single-source domain transfer learning method,the performance improved about 12.49%.The specific work is as follows:(1)By designing a mental fatigue detection experiment where mental fatigue is induced by visual search tasks,EEG,ECG and video data of 32 subjects were collected(male: 16,female: 16,age: 24.06±2.49).Firstly,verify the effectiveness of mental fatigue induced during the experiment by analyzing behavioral and EEG data.And then,compared with the direct use of subjective report fatigue level or behavior indicators like reaction time setting a threshold to get fatigue / non-fatigue label,this study integrates fatigue related characteristics extracted from the EEG functional brain network and behavior characteristics as subject’s fatigue related features,using time series clustering method to realize the dynamic labeling of data.(2)In order to improve the accuracy and the ability of noise resistance of mental fatigue detection model on the basis of portability,this study constructs a multi-modal mental fatigue detection system based on EEG,ECG and video data.The average accuracy of the multimodal mental fatigue detection neural network model built in this study is 98.26%,which is significantly improved compared with the model trained with only one or two modalities.However the detection accuracy on the unknown subjects is just 51.30%.The model cannot achieve the satisfactory detection performance.(3)In order to deal with the across subjects’ problem,the domain adaptive transfer learning method is introduced.A multi-source domain adversarial neural network is added to the model built in(2),and a source domains selection strategy based on the difference in data distribution between known and unknown subjects is proposed.The model performance validation experiments proved that the detection accuracy on the unknown subjects was significantly improved.Compared with the model built in(2),the average detection accuracy on the unknown subjects increased for 21.62%.