| Hand pose estimation refers to the acquisition of the position of human hand joints from images or videos.In recent years,applications based on hand pose estimation have emerged and have important applications in human-computer interaction,virtual reality,robotics,etc.In the metaverse system,the hand is an important channel for people to interact with the virtual world,which is even more demanding for hand pose estimation.An efficient and accurate deep learning model for hand pose estimation is a basic prerequisite for the widespread use of hand pose based applications,and an important factor affecting the performance of the deep learning model for hand pose estimation is the hand pose dataset.Currently,there is a lack of domestic research on synthetic datasets for hand pose estimation.In this paper,we construct a hand pose generator model based on the basic facts of digital twin technology and anatomy about hand size and joint motion angle limitations for generating a full class of hand pose complete datasets.The main work and innovation points of this paper are as follows.(1)Using digital twin technology to construct a hand pose generator.Using digital twin technology,a highly simulated virtual environment required for synthetic data is effectively solved,and simulated human hand movements are run in the virtual environment.In this paper,we combine the principles of hand anatomy and use digital twin technology to solve the current problems such as distortion of hand models in synthetic data sets.(2)A new hand collision detection model is proposed for the hand digital twin.The collision detection of hand motion in virtual space is particularly important.In the process of motion,the posture of fingers changes with time,and the same finger will not collide due to the limitation of joints,while different fingers,collision may occur between each finger segment,making the self-collision motion between the knuckles of the hand particularly complex.In this paper,we analyze the geometric relationship of spatial vectors in Cartesian space,calculate the distance of spatial vectors,and establish the self-collision detection model of hand’s finger segments.(3)Multi modal hand pose data are collected from multiple viewpoints,including extreme viewpoints.A large portion of images in the current hand pose dataset are acquired from the normal viewpoint,and it is found that most of the models are not robust enough to estimate the hand pose in the extreme viewpoint.In this paper,acquiring hand pose from multiple viewpoints,especially extreme viewpoints,helps to improve this problem.(4)Automatic data annotation.The real dataset is annotated by manual and semiautomatic methods,both of which suffer from manual annotation accuracy and laborintensive problems.Both methods are extremely labor-intensive when the dataset is large.The synthetic dataset is computed and labeled using forward and inverse kinematics for the hand joint coordinates,eliminating the incorporation of human errors involved in manual labeling methods.The synthetic dataset also eliminates the incorporation of device errors in the real dataset by using virtual sensors to acquire data.The synthetic data generated by the hand pose generator constructed in this paper effectively solves the problems existing in the real data.The images generated using the hand pose generator constructed in this paper are compared with NYU,Syn Hand5 M and other datasets to demonstrate the effectiveness of the constructed dataset in terms of image quality and hand pose accuracy. |
