Classification And Recognition Based On EEG-NIRS Dual-modal Motion Intention

When people observe the action of others,they naturally try to understand the intentions underlying the action.This neural mechanism is called action understanding,which has an important influence on the development of mental,language comprehension and socialization.At present,single mode measurement has been used in most studies on action understanding,which has defects in the measurement effect and the temporal and spatial resolution.This study adopted electroencephalography(EEG)and near-infrared spectroscopy(NIRS)bimodal system which can realize synchronous measurement of blood oxygen signal and electric potential signal of brain to analyze the neural mechanism of action understanding in multi aspects.Brain activity was measured using the EEG-NIRS bimodal device during the observation of these two types of hand-object interaction pictures corresponding to drinking and moving intentions.The main research work is as follows:(1)In this paper,we mainly study the frequency domain characteristics,time-frequency characteristics and common spatial pattern(CSP)classification and recognition rate of EEG signals.It was found that the extracted CSP features could achieve better classification accuracy.From the three aspects of the variance,the amplitude mean absolute value and the peak,The motion intention of the blood oxygen signal was analyzed,and it found that the two movements of the intention of blood oxygen change was inconsistent.Average absolute value of the amplitude could better distinguish between two kinds of action intention.Through the study of the characteristics of EEG and NIRS,it was found that the activation of the moving cup in the left brain area was stronger than that of the drinking water.And the activation of the drinking water intention in the right brain area was stronger,indicating that the understanding of the motion intention was lateralized.(2)In order to fully use the dual-modal signal,this study constructs the BCI system output decision before the data fusion processing.Because the two types of signals come from two different types of sensors,this study focuses on the study of feature layer fusion based on the linear discriminant analysis(LDA).Firstly,the LDA was used to classify the extracted EEG signals and the characteristics of cerebral blood oxygen signals respectively.The accuracy of single-modal cross validation was obtained.Then,based on the feature layer fusion of LDA,two types of single-modal feature information were combined to classify the model,and the accuracy of cross-validation was obtained after fusion.The LDA method was used as fusion and classifier for EEG-NIRS signal that the fusion was on the feature level and the classification was done on the feature after fusion.The average accuracy of the fusion feature method was 74.4%,which was higher than either single-mode correct rate.And owing to the data fusion technology,the subject who had little response to one brain imagery technology could be replenished in another brain imaging technology.Thus,the average classification accuracy obtained by the dual mode was 4.2%higher than the EEG single modal and 19.8%higher than the NIRS single modal.Besides,the feature fusion and classify was done at the same time with LDA,so the time of signal processing was reduced and the response speed of the system was improved.(3)In this paper,the decision layer fusion method based on error back propagation algorithm(BP)was proposed.Firstly,the EEG signal and NIRS signal were extracted by BP,and then the output of the two signals was connected in series for BP input variables to achieve decision-making integration.The recognition accuracy of BP decision-making fusion was also higher than that of single-mode,but it was not as good as that of LDA.Maybe the integration of feature layer could accommodate more information to improve the credibility of the system,and decision-making integration of the information may not contain the rich feature layer.LDA had rigorous mathematical theory,its parameters could be obtained in training;and BP was more dependent on the prior knowledge of the data set to use the appropriate parameters.BP was based on the principle of minimization of experience risk,easy to converge the local optimal solution.Therefore,using fusion data from dual modal brain imaging technology to establish BCI system improved the spatial coverage of system and reduced the degree of fuzzy information in system.