Research Of Hand Pose Estimation And Shape Reconstruction Based On Graph Convolutional Network

Vision-based 3D hand pose estimation and shape reconstruction play important roles in human-computer interaction and have a wide range of applications in many fields such as virtual reality and intelligent robots.In recent years,the rapid development of deep learning and neural networks has provided powerful technical support for computer vision.A large number of excellent vision-based hand pose estimation and shape reconstruction methods have emerged.Although these methods have achieved gratifying results,RGB images have problems such as complex background and lack of depth information,so there is still room to improve for pose estimation and shape reconstruction.To solve the above problems,this paper presents a hand pose estimation and shape reconstruction method based on a single RGB image.The specific research content can be summarized into the following two parts.(1)This paper presents a hand pose estimation method based on a single RGB image.Considering that inferring 3D pose from RGB images is a highly nonlinear problem,this paper adopts a two-stage method,that is,first estimate 2D pose,and then lift 2D pose to 3D space.Because the hand RGB image has the characteristics of complex background and serious selfocclusion,the 2D pose estimation module is built with CNN which has powerful image representation capabilities,to realize the detection of the 2D hand joint location.In view of the lack of depth information,the GCN-based 3D pose lift module is used to excavate the skeleton structure and semantic information of the hand,alleviates the inherent depth ambiguity and occlusion problems of RGB images,and realizes the mapping between 2D pose and 3D pose.In order to further improve the accuracy of the pose estimation,a pose structure discriminator module is designed,which embeds the geometric structure and motion constraints of the hand in the network learning and forces the network to produce a more reasonable hand pose.(2)The hand pose alone is not enough to achieve a complete reconstruction.Based on the pose estimation,this paper presents a weakly supervised hand mesh reconstruction method based on GCN.The difference in the way of image acquisition and the scene will cause the images from different datasets to have great diversity.The hand pose is not only irrelevant to the appearance characteristics of the image,such as the background and texture,but also has homogeneity in different data sets.Moreover,it can provide the coordinate information of the joints under the mesh,so it is a good choice to use the hand pose as the initial input of the mesh recovery network.However,there is a large gap between the sparse pose and the dense mesh.Therefore,this paper constructs a Mesh GCN network,which recovers dense hand mesh from hand pose in a coarse-to-fine step-by-step recovery strategy.In addition,dense hand mesh annotations are difficult to obtain.This paper presents a weakly supervised mechanism to provide constraints for datasets without mesh annotations to achieve effective hand mesh reconstruction.The experimental results on several public data sets verify the effectiveness of the hand pose estimation and mesh reconstruction method in this paper,and its performance is equivalent to the state-of-the-art methods.