Research And Implementation Of Continuous Chinese Sign Language Recognition Based On RealSense

As technology advances rapidly,more and more human-computer interaction methods appear in people's daily lives,and human-computer interaction methods studied with gesture recognition have come into being.Sign language,as a special gesture,is of great significance in the realisation of human-computer interaction.There are two main streams of sign language recognition technology,one is wearable device based sign language recognition and the other is visual based sign language recognition.Wearable devices are inconvenient to carry,which affects their promotion in society.Sign language recognition based on vision has become one of the hot spots for research with the advent of visual sensors such as Leap Motion,Kinect and Real Sense.In this paper,continuous sign language video data is captured with the help of the somatosensory device Real Sense D435 i,and the depth information provided by the Real Sense SDK and the tracking of the corresponding hand skeletal point positions are used to conduct a sign language recognition study.A method based on sign language key actions combined with HMM models is designed to recognise continuous sign language.The method extracts key actions from sign language videos,reducing the difficulty of processing sign language videos directly and also improving the accuracy of recognition.Firstly,the improved Cam Shift algorithm combined with the Real Sense camera is used to track and correct the sign language trajectory,avoiding the hand position bias caused by tracking directly with the Real Sense camera and also improving the Cam Shift algorithm tracking stability.Based on this,an extraction method based on the point density of the palm trajectory curve is designed to extract the key movements in continuous sign language.The hand is then segmented from the image using a depth thresholding and skin colour detection fusion algorithm,which reduces the effects of illumination and background factors,followed by feature extraction of the segmented hand image using a Hu matrix first fourth order matrix fusion image geometric features to avoid the effects of higher order matrices.Finally,the key action hand features combined with the HMM model are used to complete the classification and recognition of continuous sign language.In this paper,a continuous sign language recognition system was designed.By using experimental validation and testing of the system,the identification of 16 consecutive Chinese sign language sentences consisting of small batches of words was completed,with a recognition rate of 91.2% for those who participated in the data collection and around 83% for those who did not.