Research On 3D Non-rigid Reconstruction Of Dynamic And Deforming Object Using Monocular Video Sequence

3D(three-dimensional)reconstruction of non-rigid dynamic or deforming object from monocular video sequence refers to recovering the 3D surface structure of the target object in each frame from the 2D projection image sequence obtained by the camera,thereby obtaining dynamic model that changes with time.Non-rigid 3D reconstruction of dynamic or deforming target is an active research area and key problem of computer vision,because of its important theoretical and application value.3D reconstruction algorithm for static rigid targets has been matured and established a relatively complete system.The reconstruction of non-rigid dynamic targets has become a key issue in recent years.For robot systems or augmented reality systems,3D reconstruction of non-rigid dynamic targets in the environment is the basis and key technology for the perception and understanding of the scene.It is also a necessary prerequisite for human-computer interaction,as well as interaction between virtual and reality.At the same time,it provides a convenient and fast method for generation of virtual content or object.Therefore,this technology has important application value in many industries,such as military,machinery manufacturing,medical care,education,communications and entertainment.3D reconstruction algorithm of dynamic and deforming object has many classifications,such as marker-based method,binocular reconstruction method,multiview stereoscopic reconstruction method,multi-sensor method.Due to the limitations of sensor equipment and the various environments,monocular reconstruction system has the characteristics of convenience and wide application range.It is particularly important for some small devices with only normal camera as sensor and in some scenes with only visual signals work.However,most non-rigid dense reconstruction systems based on monocular sensor run slowly,which is far from the real-time standard.There are also monocular reconstruction schemes that are batch mode systems and all video sequence frames need to be processed simultaneously after acquisition.The problems of efficiency and non-online prevent the performance and application of monocular dynamic dense reconstruction systems.High-speed monocular dynamic target dense reconstruction system is still an open challenge.Although there are some real-time monocular reconstruction systems,they are all sparse which track only a few key points and do not generate a dense three-dimensional surface model of the object.Therefore,this dissertation focuses on the research of 3D dense reconstruction algorithm for non-rigid dynamic or deforming target from monocular video sequences.The key issues about 3D non-rigid dense reconstruction are discussed.Research on optimization of reconstruction results,improvement of the computational efficiency,enhance of the robustness,extension of the application scope,and solution for the nononline problem has been done.The main research contribution and results of this dissertation are as follows:1.A template-based 3D dense reconstruction system for non-rigid dynamic and deforming object from monocular camera video sequences was established.The timevarying 3D model of target object was obtained by using energy minimization method.The monocular reconstruction system is simple in structure,low in cost,and easy to use.The template-based reconstruction scheme provides a sequential frame-to-frame online reconstruction method with the possibility of achieving real-time performance,and also has advantages in dealing with point-based systems,multi-scale,occlusion,and blurring.Iterative Schur(IS)based linear solver has been adopted which combines preconditioned conjugate gradient algorithms and Schur complement tricks class of method to achieve faster convergence and improve reconstruction efficiency.Experiments were performed using real image sequences,including reconstruction under occlusion conditions,and the feasibility of the system was verified.2.A simple and controllable method for constructing 3D template based on monocular video sequences is constructed.This method is used as a preprocessing stage for the 3D non-rigid deforming reconstruction algorithm.It can fulfill the template construction of various complex objects such as human faces and human hands to broaden the scope of use of reconstruction methods.This stage requires a set of frames where either the object is static and the camera moves,or the camera is static and the object is moved under a rigid transformation as input.Firstly,using VisualSFM's standard structure from motion method,the camera external parameters and sparse key points reconstruction are obtained.The algorithm has high accuracy and robustness.Then,a dense depth map is calculated for each frame through a Multi-View Stereo algorithm.All-resolution depth reconstruction will not only greatly increase the processing time,but also generate more noise.Therefore,the resolution of the depth map is controlled by parameters.Finally,the data of each frame is merged to form a point cloud,and a surface mesh template is reconstructed using a floating scale surface reconstruction algorithm.Optimized matching based depth reconstruction method and floating scale surface reconstruction method improve the robustness of the algorithm under strong environmental changes,uncontrolled image acquisition,and data redundancy,making the template construction result as accurate as possible.3.The 3D non-rigid reconstruction system from monocular video sequence has been implemented in an AR system,which realizes 3D reconstruction of non-rigid deforming object as well as generation and rendering of virtual object in virtual environment.Hololens is currently the most widely used and most mature AR device.It has the function of roughly reconstructing static surrounding scenes.Currently,it does not support the reconstruction and tracking of non-rigid deforming objects.In addition,Hololens does not provide developers with direct access to raw depth sensor data,but only provides an interface of the rough 3D reconstruction mesh.In order to complete the reconstruction of deforming object in the environment,it is a feasible solution to process the Hololens camera video stream data using the reconstruction method of this dissertation.When applied to the AR system,the efficiency and reconstruction detail scale are balanced,the redundancy of the pyramid model is circumvented,and the network communication is used to separate the reconstruction system from the AR system to improve the efficiency.The corresponding experiment was conducted and the feasibility of the system was proved.