Research On Real Time Sign Language Recognition Based On Depth Information

With the rapid development of artificial intelligence and other technologies,gesture recognition has gradually become a hot issue as a major human-computer interaction.As a special gesture,sign language is also the primary way of communication for language impaired people.Sign language carries abundant information and has good expression ability.Generally normal people do not have sign language skills,so the study of sign language recognition based on computer vision can not only facilitate the communication between language impaired and normal,but also has important significance for the development of human-computer interaction.The extraction of the static hand feature,the trace of the dynamic sign language and the strategies of the real-time recognition are focused on and improved,based on the motion trajectory and key hand shape in the process of extracting sign language through the Kinect sensor.The hand shape is acquired by using the method of hand-segmentation combining depth threshold and skin-color threshold: firstly,the obtained depth data is subjected to threshold segmentation and the area of the hand and the small arm is segmented,then the area is mapped to the color image space and the skin color threshold segmentation is performed.The trajectory is acquired by tracing the coordinates of the skeletal point,and then it is preprocessed.A feature extraction method combining Hu moment and SURF is proposed: Hu+SURF-BoW.The SVM classifier adopts different features to classify static sign language,and the experiments show that the Hu + SURF-BoW feature has good recognition performance and has better stability than Hu moment and SURF.A method to remove SURF mismatched points in the process of sign language image registration is proposed: Using the main direction of the hand binary image as a reference,the relative angles of the matching feature points in the two images are determined,then the relative positions of the feature points are calculated to remove the mismatched points.Kernelized correlation filters(KCF)algorithm,a high-speed tracking algorithm,is applied to sign language tracking to track the hands.Considering scale changes and occlusion in the hand sign language process,the independent multi-scale correlation filter for multi-scale tracking is established,and then Kalman Filtering and adaptive model updating strategies are used to counter occlusion.Dynamic time warping(DTW)is used as the sign language recognition algorithm.The matching speed of DTW algorithm is improved by the conditional constraints of endpoint relaxation,early termination of matching and the candidate sequence was sorted by LB_BC.A method of extracting key hand shapes based on the density of hands trajectory points is proposed in order to improve the real-time performance of the system.Then the sign language recognition system using the recognition strategy and method is designed,it completed the recognition of 70 sign words in real time with a recognition rate of 90.54%.