Research On Multi-view 3D Hand Reconstruction Method

Virtual reality technology is becoming more and more important.3D hand acquisition is a bridge between virtuality and reality.The interaction based on the virtual 3D hand is real and natural.But at present,3D hand acquisition is not easy,and data plays an important role.The current methods have problems such as insufficient accuracy,low efficiency,or poor robust-ness,etc.,which are restricted by the 3D data of the hand.Therefore,high-quality hand 3D reconstruction is very important.Obtaining human hand model and constructing a data set with multiple data annotations are important.The main work and contributions are:(1)Designed and implemented a multi-view 3D acquisition system suitable for hand acquisition.A stable,high-precision,and expandable 3D hand hardware acquisition system is designed to solve the problems of traditional systems.First,industrial aluminum materials are used to build support structure;then a LED light source is used to construct a full-angle lighting; finally,the system is equipped with a SLR camera to acquire images.In addition,a reliable calibration method proposed.(2)A hand reconstruction method based on multi-view geometry and deformed mesh is proposed,which gives accurate multi-label.First,use the acquired 2D hand nodes and contour prior to make a energy equation for pixel matching,then optimize the disparity map and point cloud to estimate the depth robustly; then reconstruct the 3D hand nodes,and use linear blender skinning and non-rigid ICP algorithm to obtain the hand surface; finally,the details of the hand are inferred through the image texture.(3)A 3D hand point cloud reconstruction method based on neural networks is proposed,improving the robustness of reconstruction and making data collecting easier.Based on the deep features,an improvement to the traditional 3D reconstruction method is proposed.First,self-labeled training based on 3D feedback improves the feature extraction performance of the feature extraction network; then uses the graph neural network based on the attention mechanism to achieve feature point matching to estimate the camera position; finally,use the depth estimation network of the cost volume to obtain the complete hand surface.(4)Collected and release a multi-labeled high-definition hand 3D data set(Hand3DStudio4K Dataset).The data set contains more than 40 K high-definition multi-view hand image data with reliable multi-modal data annotation.The data set is beneficial to the research work of 3D hand reconstruction and industrial applications.