Research On Chinese Sign Language Recognition For Middle And Small Vocabulary Based On Neural Network

Sign language, serving as the native language of hearing impaired people,bridges the communication gap between the normal and the deaf communities.However, since sign language is still inherently arcane for general public,communication difficulty still occurs from time to time. The research of sign language recognition (SLR), whose goal is to translate sign language into text or speech automatically, would exploit an effective way to help us get rid of this inconvenience. Meanwhile, as an indispensable branch of human-computer interaction research, sign language recognition research is becoming more and more important in this intelligent age.Data acquisition and processing, the design of features, and the selection of recognition models are the three essential factors that must be considered to design an efficient sign language recognition algorithm. Microsoft's Kinect device can easily and economically captures the precise color data, space depth data and joint point coordinates. Compared to the traditional devices like data gloves or two-dimensional cameras, the Kinect device is more suitable for sign language recognition research. Therefore, in this dissertation, Kinect is used as the acquisition device of sign language data. What's more, the design of sign language feature and the construction of recognition model of sign language will be studied thoroughly.The main research contents include:1. In order to better distinguish different hand shapes, a new Specific Hand Shape descriptor is proposed. By analyzing the characteristics of hand shapes of Chinese sign language words, this dissertation proposes six ingenious designed criteria and a fast algorithm of hand shape selection in order to design a specific hand shape database. The SHS descriptor is designed based on the specific hand shape database and convolutional neural network. The recognition accuracy of hand shape based on SHS descriptor is 99.59% while the recognition accuracy of the traditional feature of histogram of oriented gradient is only 94.35%. The experimental results show that the SHS feature is better for representing hand shape.2. Based on long short-term memory recurrent neural network, an encoder-decoder LSTM model is developed to recognize the isolated Chinse sign language words. The proposed method achieves a recognition rate of 98.67%, which is better than the traditional hidden Markov model on a Chinese sign language database of 80 words.3. On the basis of the LSTM algorithm and the construction rule of Chinese sign language words, a bidirectional long short-term memory recurrent neural network is applied to extract the context information of sign words. Then, this dissertation studies a continuous sign language recognition algorithm based on BLSTM encoder-decoder structure. The proposed method achieves a recognition rate of 94.63%, which is better than the model based on LSTM on a sentence library consisting of 20 Chinese sign words.