Research On Gesture Recognition Technology Based On Deep Learning

Gesture communication is a common daily communication form and has become an important part of human-computer interaction.With the increasing demand for the accuracy of gesture recognition and the application scenarios,gesture recognition based on monocular cameras has received more attention.Due to the large deformation,self-occlusion,complex background of gestures,we explore how to improve the accuracy of gesture recognition based on monocular RGB images,using deep learning technology.In order to solve the above problems,we design a gesture recognition framework combined with hand pose estimation.The framework consists of two parts:First,we design and implement a deep learning model,HandPoseNet,to extract the 2D position of each finger joint from the image accurately.We use confidence map to indicate the position of the finger and the confidence maps are generated by HandPo-seNet for gesture images.We train on three hand pose estimation datasets to obtain a reliable HandPoseNet model suitable for multiple scenes.Then,combined with the trained HandPoseNet model,we design and implements a multi-modal deep fusion network,useing the original image and the finger gesture o'btained by the HandPoseNet model as the input of the fusion network to perform ges-ture recognition.Different from the existing multimodal method,the finger pose is au-tomatically generated by the pose estimation model,and the fusion framework essen-tially uses only the color image as the input.In order to make full use of the comple-mentarity between the two modalities,we designs two deep fusion approaches and ex-periments on several benchmark datasets.The experimental results show that the method of gesture recognition combined with pose estimation can effectively improve the accuracy.