Research On Robot Action Instruction And Evaluation Based On Brain Machine Brain Interfaces And Its Computational Impelement

In all ages,human's cognition and learning for the world are all from the information that extracts from brains.Based on the cognition of brain,we will research the brain machine reciprocity and fusion(BMR and BMF).In such BMR and BMF,the parts of artificial intelligence have strong executive power and feedback ability,while the parts of biology intelligence have strong cognitive logic.By integrating such two parts,the BMR and BMF have widely application for medical rehabilitation,military scenario,and entertainment.This research based on the general project of national natural science foundation of China:" A Brain-Machine Integrated Approach Engaging Normal People and Its Application in Robot Dance(No.61673322)".The purpose of this research is to construct brain machine brain interfaces(BMBIs)that engaging normal people based on non-invasive electroencephalograph(EEG)signals.Besides,the constructed BMBIs will be used for the robot behaviors instruction,judgment,and correction.The detail of such four research aspects and their innovative contribution are as follow:First,to solve the contradiction between the application scenes and signals sensitive and sampling rate,this paper proposes a novel EEG signals up-sampling rate and up-sensitivity reconstruction algorithm based on generative adversarial networks with Wasserstein distance(GANs/WGANs)models.The proposed models used a new calculation loss function based on multi-dimensional features fusion for the training of GANs/WGANs models.Experimental results from three datasets of different sampling rate and different sensitivity have shown that GAN model is fit for EEG signals reconstruction of different sensitivity,while WGAN model is fit for EEG signals reconstruction of same sensitivity.The proposed loss calculation function has stable iterative losses to reconstruct EEG signals with higher classification accuracy.Based on the reconstruction results,in the real-world application scenes for constructing BMBIs,we first use the high sampling rate and sensitivity EEG dataset trained a reconstructed GAN model;then,we use it to reconstruct low sampling rate and sensitivity EEG signals for applications.Second,the effect of different speed modes during AO for the human mirror neuron system(hMNS)has been explored before constructing BMBIs.Then,in the construction of BMBIs,the optimal speed mode of AO paradigm design will fully activate hMNS to obtain the best classification of brain computer interfaces(BCIs).During the design of AO paradigm,based on the humanoid robot platform to design a waving AO paradigm with four different speed modes and left and right arm performing the same waving actions.In the EEG experiments,six subjects have been invited to participant exploring experiments to extract EEG signals,and the experimental datasets are analyzed by a novel convolutional neural networks(CNNs)architecture.Experimental results have shown that the activation degree of occipital region is higher than sensorimotor region,while the robustness degree of occipital region is lower than sensorimotor region.When the AO paradigm of repeated performing action within visual refractory period of[360,2,000 ms],the improving of actions' speed will contribute to the improvement of activation of hMNS.However,if the speed mode is out of refractory period range,the activation of hMNS will be significantly decreased.Therefore,during the design of robot's action,the best method is to confirm the fastest speed within the refractory period range to obtain the best activation of hMNS.Third,to improve the performance and information transfer rate of BCIs based on MI(MI-BCIs),this paper proposes a novel algorithm for classification based on fusing spatial-spectral-sequential relationships.This algorithm first extracts filter bank common spatial patterns features(FB-CSPs);then,the extracted FB-CSPs features are incorporated into recurrent neural networks to extract spatial-spectral-sequential relationships,the fusing multi-dimensions features significantly improve the classification performance of MI-BCIs.Experimental results on public MI datasets show that suitable number of hidden layers in RNNs will significantly improve the performance of classification.Therefore,in the real application of experimental data,we must select the suitable hidden layers and use the sliding crop strategy to crop the long EEG signal to several time slices that correspond to the length of hidden layers number.Using the cropped time slices of EEG segments to fit the hidden layers number of RNNs for classification,the classification results will obtain significant improvement on accuracy and ITR,but the time complexity of classification will not significantly improve by RNNs.Based on the improvement,we can construct BMBIs based on AO-BCIs and MI-BCIs.Last,based on BMBIs,this paper chooses the actor-critic reinforcement learning system based on preference selection(RL-teacher)as the evaluation and correction algorithm for robot actions.The conventional RL-teacher suffers from a high time complexity of preference slection and the limiation of application environment.To solve such problems,thei paper proposes a novel preference selection obtaining algorithm based on ErrPs-BCIs.This algorithm applies a standard ErrPs evoked paradigm to obtain preferences from different subjects.Due to the standard experimental paradigm for preference selection,we confirm the time complexity of preference selection within a controlling range.In addition,by directly using the BCIs for RL-teacher training,the preference selection obtaining algorithm based on ErrPs-BCIs will be not of environment limitation.During the open environment,this algorithm obtains stable preference selection for the training of RL-teacher to complete the task reinforcement learning based on actor-critic.The well-trained RL-teacher then teaches the robot to correct its actions to the target actions.For this end,this paper focuses researches on EEG signals up-sampling rate and up-sensitivity reconstruction,action observation(AO)paradigm design,motor imagery(MI)classification performance improvement,and deep reinforcement learning from error-related potentials(ErrPs),respectively.The EEG signals reconstruction solves the problem of low sensitiviy EEG signals applicaition for real-world.The design of AO experimental paradigm solves the design of AO-BCIs,and the MI classification algorithm improvement helps us build the BMBIs system.Based on the BMBIs system,by using ErrPs to instruct the deep reinforcement learning sysmt,the calculation method has been used to implement robot's action instruction,evaluation,and correction based on BMBIs.