Research On Real-time Gesture Recognition Based On Vision

In recent years,with the popularity of computer equipment in people's production and life,human-computer interaction has become one of the most important areas in computer application research.Different from the traditional keyboard and mouse input methods,gesture recognition is more natural way of human-computer interaction,which has become a popular research direction in the field of human-computer interaction.Traditional gesture recognition technology mainly relies on data gloves or specific devices,which has the disadvantages of inconvenient wear,high price,and low accuracy.With the development of computer vision research and the popularity of computer camera equipment,it is necessary to study gesture recognition method based on computer vision,which has the characteristics of low cost,real-time,easy to use and high accuracy.It will not only improve the user experience of the gesture recognition system but also promote the application of gesture recognition technology in daily life.Gesture recognition is generally divided into static and dynamic gesture recognition.This paper used the theory of machine learning and deep learning to study the static and dynamic gesture recognition methods based on Leap Motion and common webcam.In the study of static gesture recognition,a feature T based on the mutual distance between the fingertips was designed for the numerical data of Leap Motion's gesture model.The experimental results showed that the feature can significantly improve the accuracy of gesture classification.In order to further improve the classification accuracy,this paper proposed a multi-feature fusion method to fuse the numerical features of the gesture model and the HOG feature of the gesture images.And the multi-classification support vector machine(SVM)method was used to perform the ten-fold cross-validation experiment in the gesture data set.The experimental results showed that the classification accuracy rate on the test set reaches 99.42%.Based on the work above,this paper proposed an effective real-time static gesture recognition framework based on Leap Motion.For dynamic gesture recognition,this paper proposed a dynamic gesture recognition network 3D-GesNet based on 3D convolution,and a series of methods were proposed to improve it.Experimental results showed that this method surpasses most mainstream dynamic gesture recognition methods.Unlike 2D convolution which can only learn the spatial features of the image,3D convolution can learn both the spatial and temporal features,which maintains the uniformity and integrity of the spatial and temporal features.The 3D-GesNet proposed in this paper only takes the RGB information of the gesture as input,and can obtain the classification accuracy of 94.59% in the large-scale gesture data set Jester.Meanwhile,the recognition speed of 3D-GesNet can reach 182 FPS and it has the capability of real-time recognition.In addition,this paper proposed an effective spatial-temporal data enhancement method for dynamic gestures.The experimental results showed that the method can improve the classification accuracy by about 2.5%.In this paper,a series of comparative experiments on 3GN features extracted by3D-GesNet were carried out based on transfer learning and t-SNE visualization.The experimental results verified that the 3GN feature is generalized,distinguishable and compact.The main contributions of this paper are as follows:(1)For the static gesture recognition task,the feature T based on the mutual distance between the fingertips was designed for the numerical data,and the multi-feature fusion method was proposed to fuse the image features and numerical features of Leap Motion.(2)For the dynamic gesture recognition task,this paper proposed a 3D-GesNet based on 3D convolution dynamic gesture recognition network,and proposed an improved structure 3D-GesNet-B.(3)This paper proposed an effective spatial-temporal data enhancement method for dynamic gestures,which can effectively reduce the model over-fitting and improve the classification accuracy.