| The task execution efficiency of the drone greatly depends on the accurate identification of the control intention and the effective interaction of the drone mission controller with the external equipment.The traditional human-machine interaction of mission controller is mainly realized through the external devices such as joysticks.The interaction mode and efficiency need to be further improved.Identification of the attentional direction of the mission controller through physiological signals such as eye movement and electroencephalography(EEG)can improve the performance of human-machine interaction system.At present,the EEG-based eye tracking paradigm has low degree of freedom,and algorithm research is limited to classification.Attention following methods still face many problems.In order to construct an attention following method that fuses multimodal data of EEG and eye movement,this thesis mainly did the following work.An eight-directional eye tracking experiment was designed.The EEG and eye movement data of 20 subjects were simultaneously collected by EEG signal acquisition equipment and eye tracker.The subjects’compliance with the experiment was verified based on the eye trajectory.The main frequency bands of EEG signals were determined by spectrum analysis.Data preprocessing was performed using methods such as re-reference,filtering and baseline correction.An eye movement direction classification model based on eye tracking feature was constructed.EEG feature extraction algorithms such as power spectral density and wavelet transform were studied.And a feature extraction algorithm RD-method(Regression-based Differential method)based on differential feature of symmetric electrodes was proposed.Pattern classification algorithms such as support vector machine,decision tree and random forest were studied.According to the EEG feature of symmetric electrodes,2-Dimension Convolution Neural Network(2DCNN)based on temporal convolution,3DCNN based on temporal and spatial convolution,and CNN image recognition method based on brain topography were studied for eye movement direction classification.The classification performance of these methods was compared by using Mann-Whitney U test.Compared with the traditional EEG signal feature extraction algorithm,the proposed RD-method and the designed CNN models significantly improved the classification performance.The joint feature extraction using RD-method and classification using random forest had the best classification performance,with the average accuracy of quaternary-classification and octonary-classification reaching 95.41%and 75.57%,respectively.A gaze position regression model fusing EEG and eye movement was constructed.It was found that in the process of eye tracking,the EEG signal feature had approximately linear correlation with eye movement distance.In accordance with this characteristic,a feature extraction algorithm of eye movement distance was designed based on the multimodality fusion of EEG and eye movement data.According to the expected distance of the subjects’fixation point with the given target and the actual distance recorded by the eye tracker,a random forest regression model was trained,and a localization model for the fixation point was constructed.The model performance was measured using the coefficient of determination(R~2).The average R~2 calculated based on the expected and actual distance was 0.83 and 0.75,respectively.An attention following prototype system of drone mission controller based on a semi-physical simulation hardware platform and the attention following method was designed.The hardware platform of the system mainly included flight simulation platform and mission control system.The software system included experiment control module,online data reading module,online data processing module and result display module.The system recognized the fixation points of the subjects by analysing EEG signals and eye movement data in real time,and the result was displayed and feedbacked through map marker and other methods.The system was tested with an average accuracy of 86.47%.Based on the EEG signals and eye movement data of the eye tracking task,this thesis studied the feature of EEG signal during eye movement following,studied the attention following method that fused eye movement and EEG data,constructed a direction classification model and a fixation point positioning regression model which were suitable for eye tracking,and built a drone mission controller attention following prototype system,which provided a new method for the effective interaction between the mission controller and the drone. |
