Realsense-based Recognition Technology Of Three-Dimensional Dynamic Finger Gestures

With the growing maturity of finger attitude recognition technology,it is widely used in various interactive scenes.In order to adapt to the new requirements of text input,such as virtual reality,super large screen and other interactive devices,air text input technology has also appeared.This thesis refers to the text input in the air for ten scenarios of 3D dynamic finger gesture recognition was studied,and finds that the current technology has problems such as poor robustness of joint positioning,long time for word recognition,etc,to further improve the recognition rate and speed of air text input method based on finger gesture recognition,and makes corresponding research and improvement for existing problems,as given in the following.Firstly,a 3D joint position estimation method based on displacement velocity is proposed to solve the problem of inaccurate positioning of finger joints caused by selfocclusion in the positioning of hand joints.Based on displacement velocity 2D target point location estimation method,this method improves the 2D state matrix into a 3D state matrix,uses the 3D joint point data collected by the visual sensor as initial data.Using Kalman fiter to calculate the displacement speed on the 3D position coordinate data,the position of the 3D joint point is estimated when the position shift occurs.Experiments show that this method reduces the joint location error when meeting the real-time requirement.Second,a mid-air text entry method based on tapping letter recognition is proposed to solve the problem that word recognition is too long to recognize in the method of text input recognition based on finger sequence.On the basis of the mid-air text entry method based on finger sequence,this method adds the recognition of tapping letters.At first,a tapping finger is detected by tapping detection,and the specific joint length and bending angle of this finger are selected as feature values.Then,tapping letters are recognized by using KNN classification algotithm.If several different candidate words are recognized because of consecutive taps,output result is the word with the highest probability calculated by the text correction model.Experiments show that the method has capability of letter recognition,and effectively reduces the calculation amount of the text correction model and improves the word recognition speed.Thirdly,on the basis of the above research,a mid-air text entry system based on finger gestures is designed and implemented.The system acquires real-time depth-image information through RealSense 3D camera and implements hand joint recognition through RealSense SDK related functions.Using the contents of Chapters 3 and 4,the problem of joint point drift is effectively prevented,and the mid-air text entry recognition technology based on finger gestures is implemented.