Research On 3D Hand Pose Estimation Method Based On Point Cloud Deep Convolutional Network

Hand pose estimation is widely used in various fields as one of the key technologies in human-computer interaction,especially in the application of virtual reality and augmented reality,which can improve user experience and achieve direct interaction between hands and virtual objects.Recent years have witnessed a steady growth of the research in 3D hand pose estimation using depth images collected by depth cameras as information input.However,due to the complexity of hand structure,high self-similarity between fingers,and severe self-occlusion,3D hand pose estimation still has problems of poor accuracy and robustness.In this paper,depth images are converted into point cloud representations,and the point cloud deep convolutional network is constructed,which directly takes the point cloud as input.Through this natural way to learn point cloud features and perform 3D hand pose estimation,it can not only make full use of the 3D information in depth images,but also avoid the data being too huge.Firstly,through the research on the structure and principle of Point Net and its hierarchical network,this paper proposes a 3D hand pose estimation method based on improved PointNet.Firstly,the bounding box localization network is used to predict the 3D bounding box and accurately crop the hand region,and then the corresponding hand point cloud is generated to simulate the visible surface of the hand,and effectively uses the 3D information in the depth image.The hand point cloud is input into the improved hierarchical PointNet to accurately perform 3D hand pose estimation.Because the hierarchical Point Net performs local feature extraction in a hierarchical manner,so the network keeps good generalization ability.By introducing a skip connection,the improved hierarchical PointNet can make full use of the features of different levels,so it can better capture the complex structure of the hand,thereby improve the accuracy of hand pose estimation.Secondly,in order to reduce the estimation error of fingertip position,this paper studies the structure and principle of PointConv.By combining PointConv with improved PointNet,this paper proposes a new 3D hand pose estimation method.PointConv regards convolution kernels as a nonlinear function of the local coordinates of 3D points composed of weight function and density function.The learned convolution kernel can be used to calculate convolution on any point set in 3D space,and can build deep networks on point cloud.PointNet and PointConv use different methods to extract the features of point cloud,so the extracted features of them can be connected to complement each other,thereby improving the accuracy of 3D hand pose estimation.Finally,this paper conducts experiments and compares with previous studies.The proposed method is verified on NYU hand pose dataset,and the experimental results show that the proposed method outperforms most of the existing methods,and the network is simple in structure,easy to train,and fast to run.