Research And Implementation Of Sign Language Recognition Algorithm Using Deep Learning Networks

Sign language is a natural language used for communication between the hearing-impaired and ordinary people.It has unique lexical and grammatical characteristics,which erect obstacles for the hearing-impaired people to engage in social activities.Therefore,sign language recognition in an automatic way can greatly improve the convenience of society for the hearing-impaired people,as well as at the same time contribute to the construction of a barrier-free society for the disabled.Sign language recognition in a broad sense refers to understanding the meaning of sign language signals.In specific sign language recognition tasks,sign language recognition is usually defined as capturing sign language signals through a computer and recognizing each gesture in the sign language sequences.Sign language recognition can be divided into two categories: isolated sign language recognition and continuous sign language recognition according to its data.The isolated data contains only one gesture in a sign language sequence,while the latter contains more than one gesture.Based on the methods of deep learning,this paper studies isolated sign language recognition and continuous sign language recognition.First,based on the framework of “3DCNN –Conv LSTM-2DCNN” proposed by the research group,the structure of convolutional neural network and recurrent neural network suitable for the sign language data is analyzed,and various network models are compared with the size of parameter and computation size,so as to select the most suitable network for isolated sign language recognition.The redundancy and attention mechanism of spatial convolution in the Conv LSTM were further discussed,and finally a variant structure of the Conv LSTM suitable for isolated sign language recognition was obtained.On the basis of reducing the parameter and computation size,the recognition rate equal to or better than other advanced methods is achieved.In order to apply isolated sign language recognition data and network to continuous sign language recognition,thereby improving the performance of continuous sign language recognition,this paper firstly proposed an adaptive time segmentation neural network to segment continuous sign language sequences into isolated sign language fragments containing only one gesture.Compared with uniform sampling,the adaptive time segmentation network can iteratively optimize the selection of sign language boundaries based on the extracted spatio-temporal features of sign language,so as to obtain the best time segmentation results according to each sign language sequence.For each segmented segment,the isolated sign language recognition technology can be used to identify each segment,and the effectiveness of the sign language recognition algorithm based on deep learning network is verified by being compared with the spatial pyramid pooling structure.The results show that the proposed continuous sign language recognition algorithm surpass the performance of similar algorithms.The sign language recognition algorithm proposed in this paper was evaluated on four datasets,including isolated sign language datasets Jester and Iso GD as well as continuous sign language datasets Montalbano and Con GD.The modules in the network architecture were tested through ablation experiments to verify the rationality and effectiveness of the designed network,the best variant design achieved 95.80% and 61.05% recognition rates and Jaccard index of 0.915 and 0.7163(the higher the better),respectively.In the comparison of end-to-end network architectures,this article has shown excellent performance and efficiency in both tasks,especially on the premise that the size of parameter and computation of the proposed Conv LSTM variant structure is only 1/8 of the original structure,and the variant achieves a satisfying result.