Hand Gesture Recognition And Hand Pose Estimation Based On Deep Neural Networks

In recent years,hand related tasks(including hand gesture recognition and hand pose estimation)become hot topic in human-computer interaction field because they are widely used in many applications such as virtual reality,computer games and sign language recognition.With the development of deep neural networks,the technology of hand related tasks becomes more and more mature.However,hand related tasks still have a long way to go for successful real-environment applications.Many previous methods of hand gesture recognition and hand pose estimation need to segment the hand out taking advantage of the depth cues provided by the depth sensors and then classify or estimate the hand gestures.However,the drawback of the process is very obvious that the subsequent tasks will be invalid if the localization fails and the two steps cannot promote each other.For solving the problem mentioned above,an effective deep attention network for hand gesture recognition and a pose detection network based on pose anchors for hand pose estimation are presented respectively.Specifically,the contribution of this article includes the following aspects:(1)A deep end-to-end CNN framework for static hand-gesture recognition based on a soft attention mechanism is proposed,which is capable of automatically localizing the hand and classifying the gesture with excellent performance.Thanks to the soft attention mechanism,we perform gesture localization in a weakly supervised manner,which does not require bounding-box or segmentation annotations in training images.Thus,the proposed method is easy to deploy in hand-gesture-recognition systems.We demonstrate the feasibility of our method by experimental results on the two different datasets(NTU-HD and HUST-ASL).Moreover,with the fusion of color images and depth cues,namely RGB-D images,the performance will be further improved.(2)A pose detection network based on pose anchors for solving the hand pose estimation is proposed.The method is inspired by the most widely used detection framework,Faster RCNN.However,we use the pose anchors replacing the original anchors based on rectangles,as a result of which,our region proposal network can output poses directly.We use clustering methods to automatically extract abstract,representative pose anchors from the dataset,rather than manually designing.In addition,we utilize Object Keypoint Similarity(OKS)to distinguish between positive and negative samples so that we can better measure the relationship between pose anchors and ground-truth.We verify the validity of our method in a dataset called LSM-HPD,which is better than the state-of-the-art method in terms of speed and accuracy.