Research On Generating Time-varying 3D Model Sequence Using Multi-view Video

With the vigorous development of movies,entertainment and game industry,free viewpoint video in which a viewer can change freely the viewpoint and the angle when receiving and watch-ing video content has gained attentions from lots of computer vision researchers for decades,and become the main direction of the next generation of video applications.As one of the multimedia technologies,it can provide users with a three-dimensional,interactive,and immersive video con-tent.So it is an important part of computer vision that based on multi-view video.The approach based on multi-view stereo is one of the most important research topics in the field of comput-er vision,and it is also an important component of the passive 3D reconstruction technique.On the basis of it,a reconstruction technique from multi-view video sequence is formed.The recon-struction of three-dimension dynamic scenes based on multi-view video sequences,known as a technique to acquire 3D model of real-world objects from multiple calibrated videos,is one of the most important techniques to generate free viewpoint video.It is also one of the most significant and challenging tasks in the field of computer vision.Reconstructing three-dimensional dynamic scene,which generates time-varying model sequence,is one of the key techniques to achieve free viewpoint video with high frame rate.However,generating time-varying model sequence with good spatio-temporal consistency is still faced with many problems.Combining computer vision,multiple view geometry,image and graphics processing,other related theories and technologies,the thesis mainly studies the reconstruction of three-dimensional dynamic scene on two aspects.On the one hand,we focus on improving the reconstruction quality of the time-varying point cloud based on the multi-view video sequence;On the other hand,we focus on improving the spatio-temporal consistency of the time-varying model sequence to obtain the high frame rate.The main research work is as follows:1.Research on improving the speed of reconstruction sparse cloudFor purposing of improving reconstruction speed,it is suitable to adopt the shape-from-silhouette approach.However,the logic is complicated in the phase of attaining the intersection in the ap-proach.It leads to the increase of reconstruction time and the decrease of robustness of the recon-struction algorithm.In terms of this problem,we present a novel method based on the contribution-weight.In the method,the contribution-weight is stored in the matrix and then traverse one time and attain the endpoint set of the visual line segment.The proposed method generates efficiently the visual hull while enhances the robustness of the reconstruction algorithm;and then improves the reconstruction efficiency of 3D dynamic scenes based on the shape-from-silhouette approach.2.Research on expanding sparse cloudFor purposing of obtaining a considerable reconstruction quality,it is necessary to mesh the point cloud generated during the reconstruction process,and then to map the texture.However,it will increase the reconstruction time,and make it difficult to reconstruct real-timely.In terms of this problem,two novel methods are proposed.One is based on the Plane-Space-Color consistency and the other is based on the spatio-temporal-contour consistency.The former is applied to the scene there exists the abundant texture,and the latter is applied to the scene there exists the lack of texture.With the two methods,the full viewpoint and quasi-dense point cloud with appearance can be generated quickly.3.Research on improving the temporal-spatial consistency of the time-varying point cloudFor purposing of improving the consistency of the time-varying point cloud,the fast recon-struction of the interpolation model between key models is still an open-ended problem.In terms of this problem,a novel method based on the spatio-temporal-contour consistency is proposed.By the method,a time-varying point cloud sequence is quickly obtained from multi-view video se-quences.And then that the fast speed and high qualities of the 3D dynamic object tracking is a good foundation for the generating free viewpoint video.4.Research on optimizing the pose estimation using the periodic data-fusion For purposing of improving the robustness and accuracy of pose estimation for smartphones,it is necessary to fuse the pose data acquired by different sensors.In the process of camera pose tracking based on visual feature or Inertial Measurement Unit,there are the numerical errors and the error accumu-lation.To solve this problem,a novel method is proposed to estimate the pose of the smartphone using the periodic data-fusion.The proposed method improves the robustness and accuracy of pose estimation for smartphones.This thesis focuses on some key issues of three-dimensional dynamic scene,and proposes effective solutions.Theoretical analysis and experimental results show that proposed solutions are effective.