Research On Gesture Sensing Technology Based On Visual Model

In recent years,the human-computer interaction of gesture interaction has been accepted,recognized and followed by more and more people.As the core of gesture interaction,gesture sensing technology is also developing and progressing continuously,including gesture sensing technology based on visual model.As a device based on visual model,LeapMotion has caused a great sensation.Its appearance provides a new way of interaction between human and computer as well as between human and environment.LeapMotion devices focus on fine-grained finger movements,making it possible to delineate finger movements.The research work of this paper is based on the joint point data collected by LeapMotion device to study gesture sensing technology.Firstly,through comprehensive and systematic analysis of gesture action,this paper models fine-grained gesture action,extracts gesture features.Secondly,based on the results of gesture motion modeling,a middleware system integrating gesture motion features is implemented to facilitate subsequent research and development.Finally,on the basis of gesture action modeling,an online sign language recognition system is designed and developed,which provides a solution to the communication problems between hearing and deaf-mute people.The contributions of this research work are mainly reflected in the following aspects:1.Gesture Modeling:Gesture modeling is divided into static gesture modeling and dynamic gesture modeling,Dynamic gesture modeling is divided into global dynamic gesture modeling and local dynamic gesture modeling.The modeling work of static gesture includes modeling of angle,modeling of inter-finger distance and modeling of finger pointing.Dynamic gesture modeling includes the modeling for the state of the hand and the motion of the hand,The modeling work for the state of the hand is the same as the modeling of the static gesture,The modeling work for the movement of the hand includes modeling of displacement,angle of rotation and direction of rotation.2.Classification Sensing:We attribute the sign language recognition problem to the classification problem of gesture actions,and solve the problems of gesture action segmentation,gesture feature selection,gesture action multi-classification and unmarked gesture action in gesture action classification.Specifically,the gesture is segmented based on the "pause" between the two gestures,the feature of the gesture classification is determined based on the gesture modeling work,the transformation method of the SVM algorithm applied to multi-classification problem is compared and determined,the problem of unmarked gestures is solved from both time threshold and increasing the negative class categories.3.System Design:We implement an online sign language recognition system based on LeapMotion sensing device.The design of the system takes into account the calibration of the effective monitoring range of the equipment and the comparison of sign language movements.The system will feedback the position information of the hand and the status information of the device to the user in real time,so as to facilitate the position adjustment of the user's hand.The system includes a real-time output module of the camera to facilitate the user to compare their gestures.We also carry out experiments on the accuracy of sign language recognition,collect gesture action data and analyze the recognition accuracy.The recognition accuracy of gesture action corresponding to sign language reaches 97.03%.