Research On Methods For Hand Shape And Pose Estimation Based On Depth Information

In information age,the demand become more and more for rising human-computer interaction technologies such as augmented reality and virtual reality and robotics.As a step in these technologies,hand pose estimation plays an important role.Recently with the popularity of consumer depth sensors and the success of deep learning,a lot of methods for hand pose estimation were proposed,aiming at locating the hand keypoints from images.Although these methods achieved impressive results,it needs further improvements for accuracy and richness of extracted information.Therefore,it is necessary to research methods for hand shape and pose estimation based on depth information.This thesis deeply researched methods for hand shape and pose estimation based on depth information and proposed two methods,the one for improving the accuracy of hand pose estimation and the other for predicting hand pose and shape at the same time.The main works are summed as follows:(1)Considering the fact that depth map is not able to adequately represent space information of a hand and has the problem of perspective distortion while voxels consumes large memory,hence this thesis chose point cloud as the data representation.To directly extract features from point cloud,this thesis designed a new point cloud processing module named lapel,which can aware the local information in input point cloud and meanwhile own strong ability to extract features.Based on lapel module,this thesis constructed lapel-Net for hand pose estimation,which extracts features using encoder-decoder,estimates heatmaps and unit vector fields with respect to each hand keypoint from point cloud,and finally recovers the coordinates of hand keypoints.To further improve the robustness of lapel-Net,this thesis proposed two techniques including controllable rotation and sampling ensemble,where the former is applied during training to augment samples while the latter is applied during inference to compensate loss of information caused by point cloud downsampling.The experimental results on several public datasets show that the proposed method is comparable with SOTA methods and obtained the lowest error on ICVL,which validates the effectiveness of proposed method.(2)MANO model is a mesh model for generating hand with real and controllable shape and pose.However,there no existing researches for estimating hand shape and pose from a single depth map using MANO.This thesis designed a base network for estimating hand shape and pose from a single depth map using MANO.Meanwhile,this thesis proposed three estimation schemes based on the base network,for exploring impacts on network performance of different supervision ways.To evaluate the quality of generated hand shape and to do detailed analysis,this thesis created a synthetic dataset following the distribution of a real dataset,including depth maps and corresponding annotations of shape and pose.Based on the experimental results on synthetic and real datasets,this thesis compared the above three schemes and summarized their strengths and shortcomings.The results show that the method that generates hand shape and pose from depth maps using MANO is feasible,and there still exits space for improvement.