Integrating Depth And Multi-Visual Sequences For Dense Indoor 3D Mapping Using RGB-D Sensors

Mapping detailed 3D maps of indoor environments is critical for applications of mobile robotics,such as indoor navigation,localization,and path planning.Simultaneous localization and mapping(SLAM)is the key technology to create a reliable 3D map by estimating the camera pose accurately,regardless of sensors.Previously,3D SLAM relied on 3D Laser scanner or stereo vision,which are either expensive or non-real time processing.With TLS technology,the obtained 3D point clouds contain detailed structural information and are well suited for scene alignment.However,TLS lacks valuable visual information,which is contained in RGB images.Although visual images can easily be captured by off-the-shelf digital cameras and their rich visual information can be used for loop closure detection,it is time-consuming to obtain dense 3D model and unreliable in dark environments and poorly textured areas.With the advantages of low cost,lightweight,highly flexible,and high quality 3D perception capabilities,the advent of RGB-D sensors(such as Kinect and structure sensor)leds to great progress in indoor mapping.It allows the capture of depth and color information at high data rates.Various 3D mapping and SLAM systems purely relying on RGB-D sensors have been proposed in recent years.This dissertation proposed an enhanced RGB-D indoor 3D mapping system by integrating depth and multi-visual sequences.First,a coarse-to-fine 3D SLAM algorithm is used for camera poses tracking,which contains three parts,key frame detection,robust camera tracking and global optimization processing.Then we novelty proposed a bundle adjustment method by integrating 2D and 3D feature points to recover the poses of RGB sequences,which subsequently can be used for far-range modeling.Based on the RGB-D datasets,a knowledge-based 3D mapping method is finally used to extract the basic components of indoor environment.Detailed researches of the dissertation includes:1.The full set of calibration method is proposed for external and internal parameters calibration as well as the relative pose recovering between RGB camera and IR camera.Based on the depth measuring principle of RGB-D camera,the theoretical error of the depth is modeling with a depth error model,which can significantly improve the depth accuracy.2.RGB-D 3D dense SLAM by integrating visual and geometric features.In order to minimize the drift accumulation and improve the poses accuracy,we propose a coarse-to-fine loop closure solution for drift-free dense RGB-D SLAM,which enhance the existed the SLAM system by three aspects.1)To reduce the accumulating drift during incremental frame-to-frame alignment,we propose a robust key frame selection strategy by sharpness constraint,correspondence ratio constraint and baseline constraint.2)To prevent the affects from error uncertainty of the 3D correspondences,3D correspondences are weighted based on the error model of depth measurement to adjust their contributions during pose tracking equation.To make sure continues tracking in untextured areas,3D line features are integrated to refine the pose tracking results,in which a novel 3D line descriptor are proposed for line matching.3)Graph optimization is used to ensure the global consistency.Appearance-based,pose-based and knowledge-based loop closure detection systems are first cooperatively used for loop constraint detection.Considerate the error uncertainty of the loop constraint,each constraint in the graph is weighted based on the residual errors from frame registration.3.A bundle adjustment optimizer by integrating 2D and 3D feature points.We intended to innovatively integrate the 3D scene generated from image-based modeling method and the 3D scene from depth images to enlarge the measure distance of RGB-D sensors.Based on a precise calibration for both of IR and RGB cameras,the relationship between the depth and RGB camera model is recovered.The poses from 3D SLAM are used as initialization in bundle adjustment processing.Two kinds of features correspondences are defined,one is 2D correspondences without depth value,and another is 3D correspondences with depth value.By minimizing the gross projection error of these two kinds of correspondences with least square method,the camera poses would be refined simultaneously.4.Knowledge-based indoor modeling with RGB-D datasets.Traditionally,indoor modeling is manually constructed with point clouds,which is time-consuming and labor-intensive.Based on the RGB-D sensors,we reconsider the modeling method by taking advantage of the interactivity during data collection and the color information from the RGB-D frame.First,the collected raw data is preprocessed with filtering and down-sampling algorithm.For room space extraction,knowledge-based plane classification method is used to filter out wall planes and horizontal planes.Wall planes are then projected to horizontal plane.The corresponding wall boundaries are obtained with line fitting algorithm and topological reconstruction method.Finally,walls,celling and ground can be reconstructed based on the ordered boundary points.For window and door extraction,only the point cloud recovered from the tagged frame is used and a color-based region-grow segmentation method is used to separate the corresponding components from the original point cloud.Then the segmented components are projected on the horizontal plane and the 2D edges are extracted by line fitting algorithms.The proposed method can extract the basic indoor components precisely and fast.We evaluated our approaches with the publicly available RGB-D benchmark provided by Sturm et al.(2012)and three sets of RGB-D datasets collected with a handhold structure sensor device.In the first part,the public RGB-D benchmark datasets containing ground truth information for camera poses in terms of time-series are used to assess the accuracy of the camera trajectory.The results are also compared with the state-of-the-art methods.In the second part,a complementary RGB-D dataset collected by structure sensor is used to access the performance of proposed SLAM system by comparing with structure sensor SLAM system.Our datasets include sequences of rooms with difference range.The absolute mapping accuracy is subsequently evaluated with point cloud obtained from TLS,which demonstrates the model generated from RGB-D sensors can achieve the measurement error less than 1cm within 50m2 range and 5cm with 70m2 range.In the fourth part,two sets of RGB-D data are used for evaluating the effectiveness of the far-range modeling,which demonstrates that the image-based modeling method can significantly enlarge the measurement distance of RGB-D sensors.To check the effectiveness of the proposed indoor components extraction method,we examine the usefulness with three sets of RGB-D datasets which have different range and detail levels.The results show the corresponding indoor components can be precisely extracted and the measurement accuracy is similar with the original point cloud.