Research On Hand Gesture Detection In Chinese Sign Language And Sign Language Recognition Based On Neural Network

Sign language is a visual language that utilizes hand gestures,hand shape changes and trajectory information to express meaning.It is the main communication tool for hearing impaired people.At present,there are many people who need to use sign language,but the popularity of sign language is poor.Sign Language Recognition(SLR)can improve this problem and provide more convenient daily life for the impaired.At the same time,hand gesture detection and recognition in sign language is an important branch of Human Computer Interaction(HCI).This research has important guiding significance for people to transition to novel and convenient intelligent interaction.Hand gesture detection and recognition methods in sign language can be generally divided into traditional methods and deep learning based methods.In recent years,with deep learning shine in the field of computer vision,researchers have fully proved that neural network is good at extracting feature,building models and intuitive training.Therefore,based on neural network,this thesis carries out the research on hand gesture detection in Chinese sign language and sign language recognition.The main content of the research include:1.In order to improve the accuracy and stability of hand gesture detection in Chinese sign language,we propose a Multi-scale Faster Region-based Convolutional Neural Network.Owing to the characteristic,such small hand region but contain rich information,and the indistinguishable gesture categories,we construct the Multi-scale Convolutional Neural Network and Inception Region Proposal Network.We test on two sign language gesture detection data sets,and our proposed method achieves the mean average precision of 93.6%and 90.0%respectively.2.Since sign language is a kind temporal task,we construct a SLR framework based on Long Short-time Memory(LSTM)which is adopted to build the encoder-decoder.According to the separability and contextual connection between the sign language actions,the model of the sign language actions unit is integrated in this framework.The spatial and temporal feature is extracted by single-channel 3D Convolutional Neural Network,and the sign language translation process from the sign language image feature sequence input to the text sequence output is implemented by the LSTM code-decoder network.Experiments show that the method can achieve a recognition rate of 98.7%on the sign language data set.3.In order to realize the detection,tracking,characterization and recognition of the RGB picture sequence of sign language,we combine the hand gesture detection module with the recognition module.We construct a SLR framework based on dual-channel 3D Convolutional Neural Network and LSTM encoder-decoder network.The framework relies on our proposed hand gesture detection model and Median Flow tracking algorithm to obtain the gesture regions.Based on the single-channel 3D Convolutional Neural Network,the dual-channel 3D Convolutional Neural Network is designed to obtain fusion features.