Research On 3D Hand Poseture Estimation Method Based On Graph Convolutional Network

Hand is an important medium for humans to interact with the objective world.Humans rely on various gestures of their hands to manipulate objects and communicate with others.Recently,a growing number of emerging technologies need to accurately recover 3D hand mesh and pose from a monocular 2D hand image simultaneously,such as human-computer interaction,augmented reality(AR),virtual reality(VR)),robots and sign language recognition,etc.Benefited from the success of deep learning in recent years,monocular 3D hand pose estimation has achieved remarkable success,where 3D coordinates of 21 hand joints canprecisely predicted from a single 2D hand image.However,it is still a challenge to estimate3 D hand mesh from a single RGB image,owing to predicting hundreds of vertices on hand mesh that is far more difficult than estimating 21 3D joint points.With the development of the parameterized hand model MANO,it is possible to reconstruct the 3D hand mesh and pose from a single 2D hand image by learning a few hand model parameters.However,this parametric hand model has an obvious defect.It is highly non-linear to learn these parameters from the 2D hand image,as there is no explicit spatial correspondence between the model parameters and the 2D image pixels,and there is insufficient spatial information to accurately regress the model parameters.In order to solve this problem,we combine graph convolutional network and MANO parameterized hand model to design a deep learning framework for 3D hand joint point estimation and 3D hand shape reconstruction based on graph convolutional network.The framework consists of three parts:(1)With convolutional neural network and MANO hand model,we design a 3D hand pose estimation and 3D hand shape reconstruction backbone network.The high-resolution network and the deep residual network are combined as a feature extraction network,which predicts MANO parameters and features with spatial information from monocular RGB images.Then,MANO model maps these parameters into 3D mesh vertices and 3D joint points.(2)According to the spatial structure of 3D hand joint points and 3D hand mesh to construct the pose graph and shape graph,we propose a novel MANO-GCN algorithm,which consists of shape parameter graph convolution and shape parameter graph convolution.The features predicted by the backbone network containing spatial information are set as the node features of pose graph and shape graph.The nodes are connected through the edges predefined in the MANO model,and the node features are propagated along the edges to learn spatial information.(3)In order to align the hand generated by the MANO-GCN algorithm to the correct position in space,we propose a concise and efficient global hand transformation method,which utilize the spatial information retained by the high-resolution network to generate hand scale coefficient,rotation matrix and translation.And use these parameters to perform a global transformation on the hand mesh and joint,so that the predicted 3D mesh vertices and 3D joint points can be accurately aligned to the image coordinate system.