Research Of Isolated Sign Language Recognition Based On Keyframe And Deep Learning

Sign language is a visual language of action,which simulates the semantics or syllables by using the changes of hand shape,gesture and spatial position of hand to express certain meanings.In addition to being an auxiliary means of normal language communication,sign language is also an important way and main tool for people with hearing impairment or deafmute to communicate and express themselves.The demand for using sign language is increasing day by day but the teaching of sign language is highly professional and the popularity of sign language is poor.The research on sign language recognition can greatly guarantee the life,study,work and other rights of hearing impaired and deaf-mute people,which has a positive social significance.At the same time,the study of sign language also has great research value for intelligent life.In this thesis we study the recognition of Chinese sign language vocabulary based on deep learning,and presents a sign language recognition method based on key frame extraction and neural network.The main work of this thesis is as follows:(1)A sign language recognition method using a pseudo-3D residuals network based on two-dimensional convolution is proposed.In view of the problem that too much computation and parameters of 3D convolution affects the network computation cost and model storage,which limits the network speed and storage.A method of combining spatial convolution and temporal convolution to replace 3D convolution is proposed,so that the network can be deepened.The residual structure is added to alleviate the effect of network deepening,so as to obtain deep information.(2)A method of sign language recognition based on keyframe cutting is proposed.In view of the problem that redundant frames in sign language video make network learning useless information leads to performance degradation.The Kinect joint information is used to judge the motion state of the hand,and the frames of the key actions are determined by analyzing the inter frame similarity model.Only keyframe features are extracted for sign language recognition.(3)An attention mechanism LSTM encoding and decoding method for sign language recognition is proposed.In view of the characteristics of sign language vocabulary that different action components have different importance in understanding meaning.Combined with the knowledge of natural language processing,LSTM is used to encode and decode the sequence features of key frames.Attention mechanism is added to realize different attention to the encoded information during decoding,so as to improve the recognition accuracy.Finally,the method in this thesis was tested on SLR＿Dataset and achieved a good result of 91.3% accuracy.