Hand Pose Estimation And Dexterous Control Based On Vision

The research of hand pose estimation is a hotspot in recent years,and in real life,it has rich applications.In addition to bring convenience and happiness to people in the pan-cultural entertainment,it can also impart operational experience from human to dexterous hand in the field of robot.Therefore,taking accuracy,real-time and robustness into account,this paper does the research on hand pose estimation and its application on controlling the virtual dexterous hand.The details are as follows:(1)This paper firstly focuses on the topic of vision based hand pose estimation from single depth map using CNN.In order to excavate more information from depth input,this paper design a new pose regression network named CrossInfoNet.The proposed CrossInfoNet decomposes the hand pose estimation task into the palm pose estimation sub-task and the finger pose estimation sub-task,and adopts a two-branch cross-connection structure to share the beneficial complementary information between the sub-tasks.The work is inspired by the multi-task information sharing mechanism.In addition,this paper proposes a heat-map guided feature extraction structure to get better feature maps,and train the complete network end-to-end.The effectiveness of the proposed CrossInfoNet is evaluated with extensively self-comparative experiments and in comparison with state-of-the-art methods on four public datasets.Based on the achievements of hand pose estimation,this paper does further study of operation migration from human to manipulator.One new kinematics mapping method is used,which integrates several conditions of each finger.The optimized results of joint angle are obtained by motion planning.Considering the domain gap between human and manipulator,this paper designs a new angle regression network with joint coordinates as transition.Its good performance is verified by the experiments.Based on the research of hand pose estimation and joint angle regression,this paper also designs two application scenes.From simple to complex,they are vision-based teleoperation system with dexterous hand only and hand-arm cooperation grasp scene.For the first application scene,this paper does the experiments in the virtual environment,which shows that the proposed mapping scheme can be applied effectively.For the second application scene,a preliminary feasibility analysis is provided,and it will be implemented in future work.