Research On Sign Language Recogniton Method Based On Convolutional Neural Networks And Recurrent Neural Networks

Sign language is an indispensable tool and communication medium with the outside world in the daily life of the deaf.But there are just a few of normal people with the sign language knowledge,which leads to a lot of difficulties in the conversation between the deaf and the normal.The mission of sign language recognition(SLR)research is using computer technology to convert the sign language movements into text,and provide a convenient platform for the conversations between the normal and the deaf.Meanwhile,SLR is a research field of human-computer interaction.SLR methods can be divided into two categories: one use data gloves and other wearable devices to collect the data of the hand,and the other is vision based method.The methods of first category simplify the process of gesture feature extraction by means of peripheral devices.Although they have high recognition accuracies,they are not very practical.And the vision based SLR methods,with the influence of illumination,background,non-specific person,etc.are not very effective.In addition,to describe the hand movement,most vision based methods define the feature by artificial.These features need to be adjusted with plenty of experience,and the effective characteristic information is easily lost in this process.In this paper,we propose a SLR method based on convolutional neural network(CNN)and Long Short-Term Memory(LSTM)network,which aims to break the limitation of device and reduce the influence of artificial-defined features.This method only uses the video image information collected by a single camera as the input data,which reduces the dependence of the method on the device.First of all,we locate the center of hand region in the video,and capture the upper body images around the center.After pretreatment,the captured image are stored as single channel input images.After that,with the strong feature extraction ability of CNN,the two-dimensional images can be transformed into one-dimensional vector directly.The vector is the input of LSTM network.Finally,the continuous sign language video is recognized by the LSTM network.CNN has achieved significant results in face recognition,object detection and so on.With CNN,we can omit the complicated artificial feature extraction in general recognition method,and exploit the information from images directly,in order to avoid information loss.Thus,CNN can sharply increase the accuracy of sign language image recognition algorithm.LSTM network,as a recurrent neural network(RNN),can utilize contextual information to guide the current state classification.Compared with traditional RNN,LSTM network can maintain long memory by using the cells and confront the vanishing gradient problem.With this characteristic of LSTM network,the interference of similar hand shapes to the continuous SLR can be decreased and the recognition rate can be improved.The experiments prove that the proposed method based on CNN-LSTM network model can keep high accuracy without external device.It can satisfy the needs of practical application.