Bimodal Analysis And Classification Of Action Understanding

When people observe the action of others, they naturally try to understand the intentions underlying the action. This neural mechanism is called action understanding, which has an important influence on the development of mental, language comprehension and socialization. At present, single mode measurement has been used in most stuies on action understanding, which has defects in the measurement effect and the temporal and spatial resolution. This study adopted electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) bimodal system which can realize synchronous measurement of blood oxygen signal and electric potential signal of brain to analyse the neural mechanism of action understanding in multi aspects. Brain activity was measured using the EEG-fNIRS bimodal device during the observation of these three types of hand-object interaction pictures corresponding to drinking, moving and unreasonable intentions. Through source localization, signal amplitude analysis and functional brain network analysis, this study located the activated brain areas and identified the differences of different action intentions. Finally, the support vector machine (SVM) was used to classify the different action intentions.This study located the activated brain areas by source localization. The results of EEG tracing showed the activation of temporal parietal lobe in the left hemisphere was strongest between 100 ms and 160 ms, and the activation of temporal parietal lobe in the right hemisphere was strongest betwwen 160 ms and 210 ms. The results of fNIRS tracing further determined the activated brain areas were located in the premotor, inferior parietal lobule and right superior temporal sulcus. The differences of the signal amplitude were analyzed by analysis of variance and t test. The results showed that the activation of drinking intention in the left hemisphere was stronger than that of the moving and unreasonable intention, while the activaton of unreasonable intention in the right superior temporal sulcus was stronger than that of drinking and moving intention. Combined with these results, this study proposed the cascade-laterality model of action understanding:firstly, the premotor and inferior parietal lobule in left hemisphere is activated and this encoding allows the individual to recognize what the agent is doing. Then the superior temporal sulcus is activated and the right hemisphere dominance for understanding why an action is carried out. At the same time, the activation of left hemisphere will be stronger when understand drinking intention, and the activation of right hemisphere will be stronger when understand unreasonable or complex intentions. In addition, the differences of the network metrics were analyzed by building brain network of EEG and fNIRS, and the results verified the laterality effect.Based on the difference analysis of amplitude and network, this study explored the classification problem between drinking and the unreasonable intention. We proposed a feature extraction method based on local properties. The local properties, which had significant difference between different intentions, were selected as the feature vector. In order to improve the classification accuracy, this study adopted the method of feature fusion to fuse EEG-fNIRS bimodal data. The average classification accuracy was 59.9%, which was higher than that of any single mode.The current study proposes a cascade-laterality model of action understanding based on the results of source localization and difference analysis, and proves that action intentions can be classified by local properties and pattern recognition.