Research Of Chinese Sign Language Recognition Technology Based On The Fusion Of Surface Electromyography And Inertial Sensors

Sign language is a kind of structured gesture and has been commonly used as the main communication manner among the deaf. However, the hearers are always not able to recognize the meanings of the sign language, which leads to communication barriers between the deaf and the hearing society. Aiming at this issue, sign language recognition (SLR) is performed to translate the information expressed by the sign gestures into text or speech based on movement sensing and computerized techniques. And the SLR also serves as one of the most natural and friendly way of human-computer interaction, with successful applications in the field of daily communication, rehabilitation medicine and virtual games.In recent years, the development of large-vocabulary oriented SLR system attracts more and more researchers’attention and has obtained a number of research achievements. Nevertheless, the number of the recognizable gestures is limited currently and still can’t meet the basic needs of daily communication between the deaf and hearing society. Meanwhile, only user-specific classification has been targeted in numerous studies whereas few efforts have been made toward user-independent classification and the recognition accuracy is relatively low. Therefore, relatively large vocabulary of recognizable gesture set with high accuracy in user-independent classification is the key problem urgent to be solved in the development of practical SLR systems.Based on the optimal classification performance of surface electromyography (sEMG), accelerometer (ACC), gyroscope (GYRO) sensors and their different combinations in the classification of three typical hand components including one or two handed, hand orientation and hand amplitude, an optimized tree-structure classification method was proposed in this paper and successfully applied in the pattern recognition of 150 frequently-used Chinese sign language (CSL) subwords. This study is regarded to be the first attempt toward relatively large-vocabulary SLR in user-independent classification based on a self-made portable and wearable wristband-like sensing system, which could lay a basis for the implementation of large-vocabulary oriented sign language interpreters. The main contributions of this paper could be summarized as follows:(1) The research on feature extraction and classification of the hand components. The data for 150 frequently-used CSL subwords were collected with our self-made data acquisition system integrating sEMG, ACC and GYRO sensors. The capabilities of various feature vectors extracted from sEMG, ACC, GYRO sensory data and their different combinations to identify each of three typical sign components including one or two handed, hand orientation and hand amplitude were explored respectively.(2) The research on SLR based on the multi-stage tree-structure classification framework. Firstly, based on the optimal feature vector of each sign-component-based classifier, an optimized 4-stage tree-structure classification method was proposed and the overall recognition accuracies of 94.31% and 87.02% were obtained for the recognition of 150 CSL subwords in user-specific test and user-independent test, respectively. Then, it was validated that the optimized tree-structure classification framework proposed in this paper could not only improve the recognition performance, but also significantly reduce the time consumption in the testing phase, as compared with the standard classification method using only multi-stream hidden Markov model. Finally, the data fusion of multiple sensory, and the involvement of GYRO particular, have been demonstrated to be effective in improving the large-vocabulary SLR performance.(3) The research on feature selection and sensor fusion in decision level. The recognition performance of different feature combinations involving ten common features extracted from sEMG sensor and 10 features extracted from inertial sensor was evaluated in the recognition of 150 frequently-used CSL subwords based on the genetic algorithm. Besides, the performance of three different decision fusion schemes including the weighted average, fuzzy integral and D-S evidence theory was also explored in the information fusion of sEMG and inertial stream model. It was found that the fusion method of weighted average achieved the best recognition performance in this paper.