Research On Chinese Sign Language Recognition Based On The Fusion Of SEMG And Inertial Information

Sign language(gesture)recognition is an important sub topic of human-computer interaction research,which can not only help the deaf people communicate better,but also greatly promote the development of human-computer interaction field.The recognition method based on portable wearable devices combining sEMG sensor with inertial measurement unit has won the attention and research of many scholars.Aiming at the difficulties of sign language recognition,location-tagging model is proposed to help the construction of sign language database.Aiming at the diversity of sign language data,a scheme of data augmentation is proposed to enhance the robustness of trained model.Finally,the recognition-verification mechanism is proposed to improve the online sign language recognition.In this thesis,we first build a sign language database which is composed of sEMG and inertia signal through myo armband.In this process,we need to solve the difficulties of positioning and labeling.In this thesis,we build a location-tagging model.The model is trained to adopt the method of transfer learning to optimize the performance of the model.In order to solve the adverse effect of sign language data diversity,we propose a random core extraction method based on time invariance and data generation mothod based the improved GAN model.By improving the time and space saturation of the samples in the sign language database,the robustness of the sign language recognition system is guaranteed.In the complex online recognition scene,the traditional segmentation-recognition mechanism has a serious problem of low robustness.This study proposes an recognitionverification mechanism,which is mainly composed of sliding window,recognition model and verification model.The sliding window sends the pulled online data into the recognition model and the verification model.The recognition model uses CNN with VGG architecture to classify the incoming data.The verification model uses Siamese Network to verify the accuracy of the recognition results.,In the optimization experiment part of this thesis,for the recognition model,the following optimization experiments are carried out,including the composition of network layer,the existence of adaptive pooling layer,the selection of activation function and the use of batch standardization.For the verification model,the optimization experiment includes the loss function of the model,model initialization and batch size.A comparative experiment between recognition-verification mechanism and segmentation-recognition mechanism is designed.The experimental results show that the performance of recognition-verification mechanism in online recognition is much better than that of segmentation-recognition mechanism.In this thesis,the key problems of sign language recognition system based on sEMG signal and inertial signal are studied.First of all,we collect sign language data through myo armband to build a sign language database.Aiming at the problem of location and tagging in the process of database construction,a high-precision location-tagging model is built.In order to increase the temporal and spatial saturation of the database and prevent the over fitting of the model,a random core extraction method based on time transfer invariance is proposed to expand the sign language data,and the GAN is improved to generate the sign language data.Finally,aiming at the problem of online recognition,an innovative recognition-verification mechanism is proposed and its key parameters and components are optimized.Compared with segmentationrecognition mechanism,this mechanism greatly improves the accuracy and robustness of online recognition system...