Real-time 3D Reconstruction Of Human-object Interactions Based On Depth Cameras

In daily life,people have a large number of complex interactions with objects in the environment.3D reconstruction of the interactive motions between humans and ob-jects is an important research field in computer vision and computer graphics.It has a wide range of applications in animation,robotics,augmented reality,etc.Currently,the reconstruction technology of human-object interactions still has many limitations in the spatial range of reconstruction,the integrity of the result and the ease of use.This disser-tation focuses on the real-time 3D reconstruction of human-object interactions based on the depth camera and proposes a series of novel algorithms to expand the spatial range of interaction reconstruction,improve the integrity of the reconstruction result,and simplify the system requirements,which provides effective new methods for the reconstruction of human-object interactions.The main innovations include:(1)A real-time method for joint 3D reconstruction of the human and indoor scene based on model segmentation.Aiming at the difficulty of joint reconstruction of large-scale indoor scene and high complexity human body in real-time,this method designs a Sigmoid function-based algorithm to decouple camera motion and human body motion.Then,this method proposes a dynamic and static model segmentation method based on registration residuals and model connectivity and establishes a hybrid voxel allocation mechanism for unified sparse voxel representation of dynamic and static models.Finally,the joint 3D reconstruction of the human and indoor scene is implemented in real-time.This method effectively extends the reconstructed spatial range of human-object interac-tions from 1.2m×1.2m×1.2m to 4.9m×2.3m×2.0m.(2)A novel method for real-time 3D reconstruction of hand-object interactions based on two depth cameras.Aiming at the complexity of hand-object interactive motions and the occlusion between hand and object,this method designs a face-to-face acquisition sys-tem and proposes a fast calibration method based on standard geometries.This method also proposes a silhouette constraint and a variable rigidity constraint for object recon-struction,and models the interactive effect between hand and object in interactions.As a result,this method simultaneously reconstructs hand poses,object geometry,rigid mo-tions and non-rigid deformations of the object in real-time,which improves the richness and integrity of the result in the hand-object interaction reconstruction.(3)A real-time method for 3D reconstruction of hand-object interactions with only one depth camera.Aiming at the heavy occlusion between the hand and the object in a single depth view,this method proposes a joint learning neural network for the prediction of 3D hand key-points and hand-object segmentation,and constructs a real dataset,which contains 3D key-points and segmentation annotations.A novel tangential contact con-straint is further proposed for object reconstruction.As a result,this method improves the robustness and usability of the system without sacrificing the quality of the reconstructed hand-object interaction.