Research On Fusion And Semantic Segmentation Of 3D Point Cloud And RGB Image

Scene recognition based on 3D point cloud data is a very challenging problem in the field of computer vision.It is also the basis for higher level visual tasks in the fields of the driverless car,machine vision,augmented reality/virtual reality(AR/VR),remote sensing mapping,etc.How to fuse RGB data with the 3D point cloud,and then improve the effect of 3D point cloud visualization and complete complex scene recognition tasks have important research significance.In the paper,we propose a method of combining RGB image and 3D point cloud to improve the quality of 3D point cloud features and thus realize the semantic segmentation tasks in 3D point cloud visualization and scene recognition.The main work of the paper are:(1)We propose a method of combining RGB information into the 3D point cloud to solve the problem of the 3D point cloud poor visualization.The development of 3D imaging technology has made the cost of 3D sensor equipment lower and lower,and the equipment has been gradually miniaturized,so it has been widely used in the industry.However,the LiDAR sensor is limited by its principle,and the information such as the texture of the scanned scene cannot be directly obtained,resulting in poor visualization of the 3D point cloud.In recent years,it has become easier to obtain RGB image data,and the RGB image has rich texture information.Therefore,we propose a method of combining RGB information into the 3D point cloud to improve the 3D point cloud visualization.The method mainly uses two kinds of data,RGB image and 3D point cloud,adopts the theory of camera imaging model,and gradually transforms the 3D point cloud from the world coordinate system to the pixel coordinate system by the camera parameter matrix,and then fuses the RGB information into the 3D point cloud.Finally,we achieve 3D point cloud visualization after fusion.This method has achieved good visualization results in the 2016 Oxford RobotCar Dataset.(2)Due to the lack of texture information in the 3D point cloud,we propose a novel deep fusion network PI-Net.The PI Net has two branches,P-Net and I-Net,which are used to fuse spatial information and appearance information of 3D point cloud.P-Net extract spatial features of the point cloud.I-Net is used to extract the appearance features of the RGB image.Finally,we use projection transformations to align and fuse the features of the two branches and achieve semantic segmentation by multi-layer perceptrons.Our proposed PI-Net has obtained good experimental results on the Stanford 2D-3D-Semantics Dataset.