Multi-stream And Multi-view Deep Learning For Surface Electromyography Based Gesture Recognition

Gesture recognition plays an important role in perceptual user interface(PUI)because hand gesture is a natural,intuitive and easy way of communication.The key issue of gesture recogni-tion is how to precisely recognize user's hand gesture from his/her input.Among various gesture recognition techniques,muscle-computer interface(MCI)has drawn great attention from re-searchers due to its wearability,insensitivity to self-occlusion and illumination condition,and capability of distinguishing subtle hand movements.With the growing demand of precise PUI,deep learning based approaches have been widely studied in surface electromyography(sEMG)based gesture recognition.On the other hand,three problems arise in existing sEMG-based gesture recognition approaches.Firstly,it has been re-vealed that only a portion of forearm muscle groups play the dominant role in a specific hand gesture,and different hand gestures have strong correlation with sEMG signals produced by dif-ferent muscle groups;Secondly,existing deep learning based approaches have shown promising success in gesture recognition using high density sEMG,but their performance is still unsat-isfactory in gesture recognition using sparse multi-channel sEMG;Thirdly,there exists large individual difference among sEMG signals collected from different subjects or data acquisition sessions,such individual difference may lead to the distribution difference between the training and the test data,which makes it hard for the learned classifier to be applied to new users or sessions.This thesis aims at proposing deep learning based frameworks to address the above-mentioned problems.The major contributions of this thesis are summarized as follows.1).A multi-stream deep leaning framework is proposed to model the correlation between hand gestures and sEMG signals originating from different muscle groups.The framework di-vides the original sEMG image into multi-stream representations,and feed them into a multi-stream convolutional neural network(CNN).The deep features learned from all streams are subsequently fused together by a feature-level multi-stream fusion strategy.Experimental re-sults show that gesture recognition performance can be effectively improved by modeling the correlation between hand gestures and sEMG signals produced by different muscle groups.2).A multi-view deep leaning framework is proposed for better performance in gesture recognition using sparse multi-channel sEMG.Multiple classical sEMG feature sets are ex-tracted and constructed into multiple views of sEMG.Then through a deep learning based view selection process,3 most discriminative views are selected and fed into a multi-view CNN.Compared to single-view learning,multi-view learning can fully exploit the information from different views,and thus brings performance improvement.3).The distribution difference between the training and the test data during inter-session or inter-subject gesture recognition is considered as a domain adaptation problem,in which the training data belongs to multiple source domains and the test data belongs to multiple target domains.To improve the performance of inter-session and inter-subject gesture recognition,domain adaptation is performed using a small amount of calibration data.