Fundamental Research On Action Understanding Based On Multimodal Brain Imaging Methods

Action understanding is a cognitive process in which people spontaneously understand the intention while observing the action of others.This cognitive process is an important part of social life and is very important for the development of mind and language.The current research on action understanding is still mainly based on the single modal brain imaging method,while this paper uses multi-modal integration of electroencephalography(EEG)and near-infrared spectroscopy(fNIRS)which reflect brain activity from neural electrical signals and blood oxygen signals to detect neuron activity.In addition,different perspectives can affect the way of action understanding.Previous studies focused on the third-person perspective(3PP)and ignored the importance of the first-person perspective(1PP).Therefore,this paper uses functional magnetic resonance imaging(fMRI)to study the effect of perspectives.The classification study of intentions and perspectives is helpful to the application of high-level brain-computer interface.As a result,this paper classifies intentions and perspective based on EEG-fNIRS and fMRI signals,respectively.In this paper,action understanding is interpreted from two aspects,which are action intention and perspective.First of all,the neural mechanism of action understanding is studied with hybrid EEG-fNIRS signal.The study of action intention is divided into three aspects and they are source analysis,differences of complex brain network and intention classification.Firstly,sLORETA and FUSION software are used to trace the source of EEG and fNIRS signal,respectively.The results of source indicate that action understanding requires the participation of the mirror neuron system(MNS)and the theory of mind(ToM),and the brain activation patterns of different action intentions are different,which means that the left MNS and ToM areas are responsible for interpreting clear intentions(drinking and moving)and the right MNS and ToM areas are responsible for interpreting unclear intentions.Secondly,brain networks are built based on EEG and fNIRS signals respectively,and the paired sample t-test is used to compare the difference of network attributes.The study found that there is a significant difference between the intention of drinking and moving in the motor cortex and MNS.Thirdly,based on the difference of network,we adopt the pattern recognition method to classify three intentions in which node attributes are features and support vector machine(SVM)is the classifier.And EEG-fNIRS signals are fused at the feature layer to improve the classification accuracy.The result shows that the average classification accuracy of EEG and fNIRS is 68.6% and 52.7%,respectively,and the accuracy of EEG-fNIRS reaches 72.7%.Next,the neural mechanism of action understanding under different perspectives is studied using fMRI.The study of perspective is divided into three parts which are brain activation analysis,brain connectivity analysis and perspective classification.Firstly,FSL software is used to perform activation on the fMRI signal.The results show that the action understanding from the first person perspective(1PP)and the third person perspective(3PP)can induce the activation of the mirror neurons.And brain activation from 1PP is significantly stronger than that from 3PP.Secondly,meta-analysis are used to introduce the key area of action observation network(MNS,ToM)and default mode network,and those key brain areas are used to construct brain network for comparing the differences between network connectivity.The result shows that the default mode network connection from 1PP is stronger that of 3PP.Thirdly,based on the difference of the networks,the node attribute of the complex brain network is used as features,and the SVM is used as a classifier to classify the perspectives.The classification accuracy based on the separate default mode network and action observation network were 76.2% and 59.5%,respectively,while the accuracy of the two perspectives can reach 83.8% with the combination of the default mode network and the action observation network.Finally,results of this paper are summarized,which are mainly divided into the following four aspects:(1)different intentions and perspectives will cause the difference of activation mode of action understanding;(2)The left MNS and ToM are responsible for understanding clear intentions,while the right MNS and ToM brain regions are responsible for understanding unclear intentions;(3)the action observation network and the default mode network participate in the action understanding under 1PP and 3PP,and action understanding under 1PP is more involved in the default mode network;(4)The fusion of EEG and fNIRS signal can effectively improve the accuracy of classification.