Data Fusion Of Velodyne LiDAR And Monocular Camera With Its Applications

Data fusion of multi-modal sensors is required to build a robust system because the individualsensor has its own limitations to extract the information and process them.Data fusion integrates the merits of the multi-modal sensors to build a robust system.This thesis is related to the data fusion of three dimensional(3D)Light Detection and Ranging(LiDAR)VLP-16 from Velodyne and a monocular camera for 3D reconstruction,two dimensional(2D)depth map,object detection and depth estimation.3D reconstruction is mainly categorized into two types indoor mapping and outdoor mapping.This thesis is related to the indoor mapping in which the research is done in an indoor environment.LiDAR can detect the distance of the object very accurately but it cannot provide the information about the color and texture of the object.2D Image is limited to only plane information such as shape,texture and color information;hence it loses the details of the third dimension due to perspective projection.This thesis can be summarized in three parts.Firstly,the development of a 3D-3D correspondence rigid-body extrinsic calibration methodology for the 3D LiDAR sensor and monocular camera.Calibration is necessary to represent the sensor data in common coordinate system.Self-made calibration pattern is used for the LiDAR-camera extrinsic calibration to generate a transformation matrix.Secondly,the range sensor and image data fusion is performed by using transformation matrix to build a 2D depth map and 3D reconstruction of the scene objects.2D depth map is interpolated to fill the empty locations due to sparse data of range sensor by using the proposed method of interpolation.Finally,the application of sensor data fusion for the object detection and depth estimation.The object detection is performed on the images by using convolutional neural network along with You Only Look Once(YOLO)algorithm.LiDAR data fused with image data is used to measure the object distance from the LiDAR.Point cloud segmentation is performed to reduce the false depth estimation due to far objects.For the segmentation of the point cloud corresponding to the detected objects is done by using kd-tree algorithm.This thesis solve the problems related to the 3D-3D correspondence based extrinsic calibration of the 3D LiDAR and monocular camera which includes the conversion of 2D image plane points into the 3D camera coordinates.3D reconstruction from the calibrated sensor fusion shows the robustness of the methodology.The proposed method of interpolation filled the empty locations in depth map with the capability of maintaining the approximate shape of the objects and it took lesser time than GPR to reconstruct the depth map,this shows that the proposed approach is robust and fast enough.Finally,the integration of the depth data with the object detection introduces the use of 3D LiDAR and monocular camera for the depth and size estimation of the objects.The depth estimation and location of the objects is beneficial for the driverless vehicles as they must identify and locate the object in world coordinate to take correct driving decision and avoid collision.