Fusion Of Panorama Based On Geometric Stitching And 3D Point Cloud

With the development of computer vision,mobile mapping,robotics,autonomous driving,etc.,how to effectively use images and 3D information of real world and 360-degree perception and mapping of surrounding environment has become a basic task.In these applications,the two most widely used sensors are cameras and lidars.The images captured by cameras have the advantages of high resolution,rich visual texture information,etc.,and lidars can obtain point cloud with scene depth and 3D geometric structure information.Because these two kinds of data complement each other,if images and 3D point cloud can be fused,3D geometry and texture information of objects' surface can be simultaneously obtained,providing 360-degree data support for tasks such as robotic localization and path planning,scene perception and understanding,target recognition and tracking,etc.In this paper,3D laser scanner and multi-camera rig are used as experimental tools to research panoramic stitching based on photographic geometry of multi-camera rig and the fusion of dense 3D point cloud and panorama.Because the intrinsic and extrinsic parameters reflecting photographic geometric relationship of each camera are unknown after installing multi-camera rig,the calibration for multi-camera rig is previous work of panoramic stitching and fusion.On the basis of comparing and analyzing the characteristics of chessboard calibration method and traditional photogrammetry calibration method,for the purpose of ensuring calibration accuracy,this paper uses the multi-photo space resection to calibrate multi-camera rig.For the distortion of wide-angle and short-focus lens,the radial tangential model is used to simulate camera's distortion.The distortion coefficients and other intrinsic and extrinsic parameters is regarded as the undetermined parameters,which is solved once by the multi-photo space resection.The original distortion images are corrected by using calibrated distortion coefficients,and the panoramic mapping model is established by using intrinsic and extrinsic parameters of multi-camera rig.The corrected images are mapped onto the panoramic sphere,and finally panorama is obtained by unfolding panoramic sphere.The fusion of image and point cloud essentially is to solve extrinsic parameters between two sensors' frame,including a rotation matrix and a translation vector.In this paper,the initial extrinsic parameters are obtained by the geometric constraint established by a chessboard,then LM algorithm is used to iteratively optimize the target equation to obtain globally optimized extrinsic parameters.This paper researches the fusion method which uses the close-range photogrammetry calibration laboratory as the medium.Firstly,the initial extrinsic parameters of laser's frame relative to calibration laboratory are obtained by ICP algorithm,and then the global optimal solution is solved by iterative optimization using LM algorithm.The extrinsic parameters of camera relative to calibration laboratory are obtained through the spatial resection.Due to the limitation of spatial position in the process of multi-camera rig's integration,the optical centers of six monocular cameras cannot completely coincide,and images are captured by multiple cameras with different angles of view from scenes with different depth differences,resulting in there may be slight misalignment and ghosting in overlapping areas.In view of this,this paper uses the depth information of point cloud to optimize the overlapping area problem to achieve the purpose of optimizing stitching effect.