Research On Methods Of Multi-View Stereo 3D Reconstruction

Multi-view stereo(MVS) aims at accurate reconstruction of 3D geometrical shape from a set of calibrated 2D multi-view images which is a fundamental yet active research area in computer vision. With the ubiquitous use of modern digital cameras, drones and smartphones, a large number of images could be easily captured in our daily life. MVS provides a promising way to reconstruct both indoor and outdoor scenes from multiple view images and plays a critical role in many practically important vision applications,e.g., entertainment, augmented reality, digital cultural relic, urban reconstruction, 3D printing, object detection and recognition.Despite the steady progress in MVS reconstruction, many existing methods are still limited in several aspect, including:(a) poor reconstruction quality in recovering deeply concave and thinly protruding structures, and sensitivity to initial conditions;(b)difficulty in recovering sharp features, fine-scale details, and high memory requirements;(c) expensive computational cost and slow convergence;(d) poor visual feedback for real-time application. To address these issues, in this doctoral thesis, several methods with novel contributions are proposed to address each of these problems.Firstly, to improve the reconstruction quality in deeply concave and thinly protruding structures, a two-phase optimization method for generalized reprojection error minimization(Tw GREM), where a generalized framework of reprojection error is proposed to integrate stereo and silhouette cues into a unified energy functional. For the minimization of the functional, we firstly introduce convex relaxation and convex optimization on3 D volumetric grids to generate an initial surface. Then, the resulting surface is refined based on triangle mesh-based surface evolution to obtain a high-quality 3D reconstruction.Comparative experiments with several state-of-the-art methods show that the performance of Tw GREM based 3D reconstruction is among the highest with respect to accuracy and efficiency, especially for data with smooth texture and sparsely sampled viewpoints.Secondly, to recover fine-scale details and sharp features while suppressing noises and robustly reconstruct regions with less textures, we present a Detail-preserving and Content-aware Variational(DCV) multi-view stereo method, which reconstructs the 3D surface by alternating between reprojection error minimization and mesh denoising. In reprojection error minimization, we propose a novel inter-image similarity measure, which is effective to preserve fine-scale details of the reconstructed surface and builds a connection between guided image filtering and image registration. In mesh denoising, we propose a content-aware p-minimization algorithm by adaptively estimating the p value and regularization parameters. Compared with conventional isotropic mesh smoothing approaches, the proposed method is much more promising in suppressing noise while preserving sharp features. Experimental results demonstrate that our DCV method is capable of recovering more surface details, and obtains cleaner and more accurate reconstructions. In particular,DCV achieves the best results on the Middlebury dino ring and dino sparse datasets in terms of both completeness and accuracy.Thirdly, to efficiently reconstruct high-quality 3D shape,we propose a simple, efficient and flexible depth-map-fusion-based MVS reconstruction method: Co D-Fusion.The core idea of the method is to minimize the anisotropic or isotropic T V + L1 energy functional using the coordinate decent(Co D) algorithm. Co D performs T V + L1 minimization via solving a serial of voxel-wise L1 minimization sub-problems which can be efficiently computed using fast weighted median filtering(WMF). WMF is then extended to larger neighborhood to suppress reconstruction artifacts. The results of quantitative and qualitative evaluation validate the flexibility and efficiency of Co D-Fusion as a promising method for large scale MVS reconstruction.Fourthly, to achieve an efficient, robust and low-cost MVS reconstruction for realtime applications, we proposed a GPU-based real-time reconstruction system with novel algorithms by using a single RGB camera. The camera we used can be a cheap webcam plugged into the computer or the camera used in smartphone or tablet. The proposed algorithms can robustly track the camera pose, then estimate and fuse the depth maps into a high-quality 3D model while refining the camera poses in real-time. Our algorithms are scalable, since complexity of depth estimation is independent to depth search range by using a random search method to initialize search space of the depth. The algorithms perform robust on the low texture region by using a confidence-based depth adjustment,in which a piecewise linear model is built based on high texture region and use to predict the depth of low-texture region. Thanks to the real-time feedback of our system, the user can freely adjust his or her scanning path in the scanning procedure.In summary, the work of this thesis not only improve the MVS reconstruction to a new state-of-the-art, but also might shed light on developing new MVS or other 3D reconstruction methods.