Research Of Human-computer Interaction Technology Based On Multi-modal Biopotentials

Biopotential based human-computer interface(HCI)is a novel human-computer interaction technology,which utilizes bioelectrical signals to provide a connection channel with external devices directly.The widely used bioelectric signals are EEG signal and surface electromyography(sEMG)signal.The EEG signal records from the scalp,which does not depend on muscle tissue.It has the advantages of fast response,safety,convenience,non-invasiveness,etc.The sEMG signal has a larger amplitude and more stable than EEG signal.It can immediately reflect muscle activity and motion intention,and the operation is convenient and natural.In this study,the single-mode signal processing method is thoroughly analyzed,and the multi-modal biopotential based HCI is deeply studied.This study proposes a novel fewer-channel common spatial pattern(FCSP)algorithm and a correlation of distance metric algorithm(Corr-DM)based on all sample pairs in a batch.The combination of these two algorithms is used for channel/feature extraction and selection.Combined with deep metric learning,we also propose a multi-modal signal processing algorithm,i.e.MWP-EMG-EEGNet,which is based on finding the maximum-weight perfect matching in a bipartite graph.These algorithms can be used for a variety of tasks to test the classification performance.Among them,our work focuses on single-trial identification performance of multi-modal biopotentials induced by ERP paradigm.The relevant conclusions can be applied to the research of reducing reaction time.The results verify that the bioelectrical signal can obtain higher response accuracy in a shorter period.Based on the above researches of these algorithms,this paper also builds online human-machine interface systems to realize online applications.In this thesis,wearable devices are designed to record the sEMG signals induced by the upper arm muscle tissue during four wrist activities.The 42-dimensional feature vector is extracted from the time,frequency and time-frequency domains.The proposed distance metric(DM)algorithm uses all samples to redefine the inter-class distance and the intra-class distance.The ratio between these two distances is used to measure the separability.Four different classification algorithms are used to evaluate the subjectuniversal feature subset obtained by the DM algorithm.The online task is to collect real-time sEMG signals through a wearable system,and to manipulate a telecar through four different paradigms to complete the specified path with simple obstacles.Online performance is tested by travel time(TT)and recognition rate(RR).The SNR of our designed FPC based online acquisition system can reach 68.91 dB.Meanwhile,our proposed DM algorithm can achieve 96.77% classification performance using 23-dimensional features.Online results illustrate that the state-machine paradigm with a 125 ms window has the highest manoeuvrability and is closest to real-life control.Subjects can accomplish online sessions by this paradigm with average TT of 48.08 s.A critical difficulty in applying the brain-computer interface system in daily life is how to reduce the number of electrodes.Therefore,our study proposes a fewer-channel common spatial pattern algorithm and a simulated annealing strategy based correlation of DM(Corr-DM)algorithm to select a universally optimal channel combination for each subject.When subjects imagine mental arithmetic and spatial rotation,we acquire EEG signals to verify algorithms.The combination of these algorithms needs only 7 channels to distinguish the above two cognitive-behavioural imagery tasks,achieving 90% binary classification accuracy.Then,the truncated weighting algorithm is coupled to determine the universally optimal channel combination.The effectiveness of this channel subset is verified by the correlation and separability in the training group.The correlation analysis result shows that the channel combination is correlated with all channels significantly.Separability analysis result indicates that there are significant differences between EEG signals of the above two cognitive-behavioural imagery tasks recorded by this optimal channel combination.The cross-subject analysis is performed on the test group.The result shows that the average classification accuracy adopting the universal channel combination can reach 93.18%.Based on the processing algorithms and results of EEG signals and sEMG signals,the shortening of reaction time by using multi-modal bioelectrical signals is studied.In this paper,the experimental paradigm based on the reaction time is designed.The CorrDM algorithm selects 10 channels.The EEG and sEMG signals are analyzed by sliding window and extended time window method separately.Compared with the reaction time of actual mouse-click,the reaction time of EEG and sEMG signals is shortened by 159.04 and 75.22 ms,respectively.The bioelectrical signal can obtain a higher recognition accuracy of the reaction motion in a shorter reaction period.Using the artificial neural network to analyze 0~400 ms EEG signals,the response recognition accuracy(single-trial ERP classification accuracy rate)can reach 93.39%.Manual feature engineering and the SVM algorithm is used to analyze 0~400 ms sEMG signal,and the accuracy of reaction motion is 88.65%.Compared with the accuracy of actual mouse-click,the accuracy of two biopotentials is improved by 60.2% and 55.46%.According to analysis results of the single-mode reaction time,our paper proposes a novel multi-modal biopotentials processing algorithm,i.e.MWP-EMG-EEGNet.The core idea of the algorithm is to find the Maximum-Weight Perfect(MWP)matching in a weighted complete bipartite graph.For each anchor sample in a training batch,the MWP matching can discover the optimal hard positive and negative samples,which can alleviate overtraining and sample imbalance.Moreover,our thesis further formulates a batch-wise loss objective based on the proposed MWP matching for deep metric learning.Our proposed loss can be trained with binary cross-entropy loss in an end-to-end manner.MWP-EMG-EEGNet can combine the advantages of single-mode analysis,which ensures the faster response time(307.22 ms)of biopotentials and obtains better accuracy of the reaction motion.The specific experimental results can be summarized as follows.(1)During the period of 0~400 ms,the multi-modal processing algorithm can achieve 96.38% recognition accuracy,which is 63.19% higher than the accuracy of actual mouse-click response.(2)Joint training of MWP and binary crossentropy loss objectives can consider separability of inter-class samples and compactibility of intra-class samples simultaneously.It can optimize the distribution of single-trial samples in the feature space.(3)The MWP matching can weight hard samples,which boosts the convergence rate and recognition performance.Based on the above offline data analysis,our thesis designs two online human-machine interface systems for multi-modal bioelectrical signals.The first system contains NeuroScan,BCI2000 and FieldTrip toolbox.The same group of subjects participated in the offline experiments are invited to perform two online experiments.The parametric model of MWP-EMG-EEGNet in the first online operation is trained only by all offline data(i.e.,N2&P3_Offline).Moreover,the parametric model in the second online operation is trained by the offline and first online data(i.e.,N2&P3_Online).The average classification accuracy of these two online experiments is 94.62% and 97.16%,respectively.However,the first system still has a portability problem.Therefore,a low-cost,portable and multi-model biopotentials acquisition system is designed.The corresponding software platform is proposed for multiple operating systems and programming environments,which can realize online data acquisition and graphical visualization interface.This system is also applied to the virtual wheelchair operation.Our methods(i.e.,designed system and algorithms)can identify the steering and forward states accurately.Moreover,in the steering state,the specifical left and right turn can also be recognized.