| The high-precision and high-density three-dimensional geometric structures of urban scene reconstructed from image dense matching is an important infrastructure data for serving or building the smart earth and smart city,which is also the prerequisite for urban planning management,disaster prevention and emergency response,resource and environmental monitoring,cultural heritage,autonomous driving,and so on.In recent years,with the rapid development of sensor hardware and oblique photogrammetry technology,the image dense matching technology has been widely used for large-scale urban scene 3D modeling.Accurate and efficient reconstruction of the 3D geometric structures of urban scene has been one of the hot issues in the fields of photogrammetry and computer vision.Compared to the Li DAR or structured lighting scanning technology,the dense matching methods are a low cost way,and have been proven to be competitive in terms of accuracy.Moreover,the texture information can be simultaneously obtained.Currently,dense matching methods still have many challenges in terms of accuracy,efficiency,integrity and automation,particularly suffer from the mismatching in noise,poor texture,repetitive texture,and occlusion areas.The traditional fronto-parallel condition is not sutiable in the urban scene consisting of many complex objects,which hinders directly generating a high-quality depth reconstruction results from dense matching.Urban scene often contains a large portion of slanted-plane-structure objects,and even non-planar structure targets can be expressed by a series of local slanted planes,but these prior information are usually missed in traditional dense matching algorithms.Although the continuous dense matching method effectively overcomes the metioned issues by introducing the slanted plane geometry model,its accuracy and efficiency still need to be further improved.Firstly,the continuous dense matching approach initilizes each pixel's plane label by randomly sampling,which is difficult to ensure that a sufficient number of pixels are assigned the initial plane labels close to the ground truth.Therefore,a lot of iteration overhead is needed to ensure the convergence of the dense matching approach,and the results of the depth reconstruction is easy trapped in local minima simultaneously.Secondly,the robustness of the matching cost function in the poor and repetitive texture areas still needs to be improved.Movreover,the framework of the current energy function optimization is sensitive to the penalty factor of smooth term,often leads to detail loss of the scene geometry structure,and the efficiency of the energy function optimization is low,which is also difficult to implement parallel acceleration.In this paper,we made a deeply research on the above mentioned issues and the key technical difficulties for the depth reconstruction of image-based dense matching,and propose slanted-plane-structure based dense matching approach,which can obtain high-precision and high-density 3D geometric information for urban model reconstruction.The details are as follows:(1)Considering the urban scene contains a large portion of slanted plane structure objects,this paper poposes HPM-TDP,which is an efficient hierarchical Patch Match(HPM)depth reconstruction approach using tree dynamic programming(TDP).HMPTDP replaces the traditional fronto-parallel condition with slanted disparity plane model to avoid the stair-casing artifacts of the reconstructed slanted object surfaces;the HMP framework is used to optimize the global energy function to overcome the shortcoming that traditional discrete energy function optimized method is hardly utilized to solve the continuous energy optimization problem,and significantly improves the accuracy and efficiency of global energy function optimization.HPMTDP first constructs the image pyramid stereo pairs,and segements each layer's reference image into a non-overlapping superpixel set to guarantee all pixels within each superpixel have the same disparity plane.Secondly,the HPM strategy is used,which employs the low-resolution disparity plane map as prior to initialize each pixel's disparity plane of high-resolution image,to reduce the iteration overhead and avoid the generated disparity plane map being trapped in local minima.In order to improve the accuracy of depth reconstruction in noise,poor texture and repetitive texture regions,the multi-resolution cost aggregation strategy and cross-based local multipoint filtering(CLMF)are integrated into superpixel-level to enhance the robustness of matching cost function in these challenging image areas,and significantly reduces the mismatching results of depth reconstruction.Finally,the superpixel-level Patch Match strategy is employed,which converts the global energy optimization into superpixel-level energy optimization problem,and the tree dynamic programming(TDP)optimizer and local expansion moves(LEM)strategy are effectively combined to minimize the superpixellevel energy function optimization problem,to guarantee that the reconstructed depth result is insensitive to the penalty factor of the smoothness term,and preserves more details of objects.(2)In view of the limitation of HPM-TDP,the semi-global Patch Match multi-view dense matching approach based on slanted depth plane model is proposed to solve the sensitivity of the image occlusion and difficulty to the parallel acceleration.The slanted multi-view depth plane geometry model is used to overcome the effect of the traditional fronto-parallel condition.The HPM framework is also successfully introduced into multi-view dense matching to improve the accuracy and efficiency of global energy function optimization,and avoid the reconstructed depth results being trapped in local minima.In addition,an effective superpixel graph structure that is suitable for parallel acceleration is proposed to improve the computational efficiency of global energy function optimization,which utilizes the four color theorem to split the superpixels into a series of superpixel subsets,and ensure that there is no intersection between any two superpixels in the same subset,then the global energy function optimization can be performed in parallel for all superpixels of the same subset.What's more,the semiglobal matching(SGM)optimizer and local expansion moves(LEM)strategy are effectively combined to minimize the superpixel-level energy function optimization.It is worth noting that the independent of SGM cost accumulation in all direction can be easily implemented to parallel acceleration during the energy optimization.(3)The performance of the proposed dense matching approaches is verified by different testing image datasets from the large-scale urban scene aerial images,oblique images to the high-resolution ground close-range images.Through the quantitative accuracy evaluation and qualitative analysis of experimental results,the validity and reliability of the proposed dense matchingapproaches are verified.Three benchmark datasets—the Middlebury 3.0,KITTI 2015,and Vaihingen datasets—were used to test the performance of HPM-TDP.The comprehensive experimental results demonstrate that HPM-TDP obtains a good performance on all datasets in terms of the(“Out-Noc”,“Avg-Noc”,“Out-All”,“Avg-All”)of(15.45%,4.16 px,24.26%,12.14px)and(5.46%,1.20 px,6.55%,1.54px)for Middlebury 3.0 and KITTI 2015 training datasets,and the(“Out-All”,“Avg-All”)of(26.32%,4.04px)for Vaihingen dataset,respectively.Three different benchmark datasets—the Fountain-P11,Vaihingen,and Zurich datasets—were used to test the performance of MHPM-SGM.The comprehensive experimental results demonstrate that MHPM-SGM obtains a good performance on all datasets.And for Fountain-P11 dataset,the pixel percentage of MHPM-SGM metho with depth error less than 2cm and 10 cm is 82.9% and 98.7%,respectively.For Vaihingen dataset,the root mean square error and average evaluation error of MHPM-SGM are(0.982 m,0.659m)and(0.871 m,0.531m)respectively.At the same time,the efficiency of MHPM-SGM can be improved 4 to 5 times by using the parallel superpixel graph structure and Open MP technology. |
PDF Full Text Request