Key Technologies For 3D Reconstruction Based On Heterogenous Stereo Vision System

3D reconstruction is the core content of computer vision.3D reconstruction based on binocular stereo vision is the most commonly method of 3D reconstruction.In recent years,many researchers have proposed a heterogeneous stereo vision system which combines a single-view catadioptric panoramic camera and an ordinary perspective camera.Compared with the conventional isomorphic binocular stereo vision system,the main advantages of this type of vision system include: it is able to acquire panoramic images of the monitored area and local high-resolution information simultaneously.Two cameras can work together which aims to obtain the depth information.To utilize this advantage,this paper extends the 3D reconstruction task to heterogeneous stereo vision systems.Omnidirectional images can provide pixel information with a 360-degree field of view for 3D reconstruction tasks.Conventional images can provide high-resolution pixel information with local viewing angles for 3D reconstruction tasks.3D reconstruction using heterogenous stereo vison system is beneficial to its application in the field of monitoring and scene understanding.This article aims to study the key issues in the process of 3D reconstruction using heterogeneous stereo vision systems.The main technology involves automatic rectification of heterogeneous stereo vision system,quality enhancement of omnidirectional image,depth map inpainting and foreground segmentation.The main contribution of this article are summarized as follows:Firstly,an automatic rectification algorithm is proposed for dynamic heterogeneous stereo vision systems to solve the problem of the relative position between the catadioptric camera and the traditional perspective camera being not fixed.Different from the traditional Hartley stereo rectification algorithm based on the eight-point method,the algorithm in this paper first unwarp the omnidirectional image to unify the imaging method of the heterogeneous image pair,and then the resolution normalization algorithm based on zero-fill is used to normalize the resolutions of two images.To solve the problem of intensity difference of pixels in two images,the feature point detection algorithm which is commonly used in remote sensing images is adopted detect feature point pairs.To improve the accuracy of essential matrix,we proposed a normalized matrix optimization algorithm.Finally,the rotation matrix R and translation matrix T are calculated by singular value decomposition.The accuracy and effectiveness of the algorithm in this paper are verified by the comparison experiment in the real environment and the simulation environment respectively.Experimental results show that the algorithm realizes automatic rectification of dynamic heterogeneous stereo vision system.Compared with the latest correction algorithm,the correction accuracy of this algorithm is improved by 34.78%.Secondly,the success rate of stereo matching is very low due to the qualitative heterogeneity between panoramic images and conventional images.In order to improve the quality of the panoramic image and reduce this heterogeneity,the defocusing blur algorithm for the panoramic image and the super-resolution reconstruction algorithm for the heterogeneous stereo image are proposed respectively.First,an omnidirectional image defocusing blur algorithm is proposed.In this paper,we directly learn the end-to-end mapping between the defocused blur image and the image after defocused blur removing.Experiments show that the proposed defocusing blur algorithm for omndirectional images can avoid the ringing effect and restore the high-frequency information of the image correctly.The peak signal-to-noise ratio of the image after defocusing blur is increased by 2.89 d B on average.To further improve the quality of the omnidirectional image,this paper proposes a variational Bayesian super-resolution reconstruction algorithm.The main purpose of the algorithm is to reconstruct low-resolution panoramic images based on high-resolution conventioanl images with limited viewing angles.The key technique of this algorithm is to use the improved full variational prior distribution to establish the prior relationship between the local high-resolution image and the local low-resolution image corresponding to the viewing angle.We use the Kullback-Leibler divergence to acquire the posterior distribution of the high-resolution image.The high resolution omnidirectional image is calculated by the mean value of the posterior distribution.The proposed algorithm achieves an improvement of up to1.39 d B PSNR and 0.003 SSIM over the state-of-the-art super-resolution algorithm.Based on the improvement of omnidirectional image quality,the success rate of stereo matching increases by an average of 25.67%,which directly improves the accuracy of 3D reconstruction and the success rate of obtaining point clouds.Third,to avoid the influence of the holes in the depth map on the acquisition of 3D point cloud,this paper proposes an improved second-order smoothness prior algorithm for the depth map inpainting.The algorithm is improved from three aspects based on the original second-order smoothness prior method.To classify the holes into edges and internal structure,the edge detection is performed on the color image.And the edges obtained by edge detection are divided into two categories by the similarity judgment equation.One part belongs to the edge of the internal structure,and the other part belongs to the edge between different regions.The missing pixels in the internal structure are filled using the original energy function.The missing pixels belonging to the edges between different regions are solved using a redefined energy function.The energy function is redefined in two ways.First,the energy function redefines the weight in the smooth term and uses the pixel gradient in the edge detection result to define the weight.Second,regular terms have been added to the new energy function.The purpose of redefining the energy function is to enhance the retention of details of edge pixels in the depth map,which improves the accuracy of depth map estimation.Experiments show that the depth map after inpainting can retain the edge structure and details precisely.Compared with the state-of-the-art depth map inpainting algorithms,the median error of this algorithm is reduced by 5.1%.The root mean square error is reduced by 11%.The average absolute error is reduced by 12%.The completed 3D point cloud data can be acquired by using the inpainted depth map.Finally,to achieve foreground 3D reconstruction by using the heterogeneous stereo vision system,this paper proposes a foreground segmentation algorithm.The algorithm combines a segmentation algorithm based on region growth and a segmentation algorithm based on edge detection.The main advantage is that it can achieve complete automatic segmentation without the user's interactive behavior and establishment of complex background model.The algorithm first obtains the seeds of the foreground and background through morphological operations and edge detection.Then,an objective function based on the seeds is established.Finally,the optimal foreground segmentation of the image is obtained by minimizing the objective function.Experiments show that the algorithm can separate the dynamic foreground target from the background accurately,and realize the individual 3D reconstruction of the foreground.The effectiveness of the above algorithm is systematically verified by the performance of the 3D reconstruction experiment of indoor environment and outdoor environment.