| In recent years,with the rapid development of sensor technology,computer science,communication engineering,network engineering,artificial intelligence,automation and other technologies,autonomous driving technology has gradually become a research hotspot,and also appears in daily life.In order to adapt to the complex and variant environment,and to perform different tasks efficiently,the multisensor fusion based scene perception has become a necessary function for an autonomous driving vehicle.However,there still exist problems of multisensor based information acquisition and usage for current autonomous driving technology,all of which affect the intelligence of an autonomous driving vehicle.On account of the multisensor fusion based scene perception technology for autonomous driving vehicles,this thesis focuses on the key technologies in data acquisition,semantic segmentation and road detection.The main contributions are listed as follows:1)To obtain pixel-wisely aligned and temporally synchronized RGB and thermal frames together,a novel paradigm is proposed to build a hybrid RGB/thermal camera.Compared with the existing schemes,five differences are introduced to make it more compact in dimension,and thus more practical and extendable for real-world applications.The first is a redesign of the structure layout of the hybrid camera.The second is a removable adjustment system to detach the hybrid camera.The third is on obtaining a pixel-wise spatial registration of the thermal and RGB frames by a two-step alignment.The fourth is to weaken the ghost phenomena from the rolling-shutter-based thermal camera.The fifth is on extending one single hybrid camera to a hybrid camera array,through which wide-view spatially aligned thermal,RGB and disparity images can be obtained simultaneously.The experimental results demonstrate the accuracy of the hybrid camera and the effectiveness of the designed stereo wide-view hybrid camera system.2)There are few researches for RGB-thermal based road detection and semantic segmentation,while the current fusion-based algorithms often ignore the unique information from a single sensor.Two new cross model based fusion methods are proposed to demonstrate the advantages by fusing RGB-thermal images.First,a middle fusion based model is built,where the output feature maps of encoder branches from RGB and thermal images are directly concatenated into a single fusion branch as the decoder.Next,the originally discarded layers after fusion operation for both RGB and thermal branches are recovered as the mimic branches to imitate the distributions of the fusion outputs,which constitutes an extended cross model(ECM).Moreover,the outputs of mimic branches at different scales are also used to imitate the corresponding outputs in the fusion branch,called a hierarchical cross model(HCM).Because there does not exist any dataset for RGB-thermal based road detection,a new RGB-thermal dataset is provided.In this dataset,the hybrid images are acquired by an optically aligned hybrid imaging device,consisting of a thermal imager and an RGB camera to output pixel-wise registration of thermal and RGB frames.The experimental results on both our RGB-thermal road detection dataset and the public RGB-thermal semantic segmentation dataset demonstrate the effectiveness and efficiency of our fusion strategies.3)Aiming at the problem that the approach only using LiDAR or RGB camera is hard to obtain accurate results for road detection while susceptible to noise,a method of the LiDAR-Imagery row-and column-scanning with image guided diffusion is proposed for road detection.First,the original point cloud from LiDAR is re-organized in an ordered way to generate a LiDAR imagery.Then the flat region is extracted from the LiDAR imagery as the candidate road region.Next,a strategy of row-and column-scanning is conducted in the LiDAR imagery to detect a finer road region from the candidate region.To fuse the point cloud with image information,the point cloud that corresponds to the above detected road region is transformed to the image space according to the calibration parameters between the LiDAR and camera.Then,two image-guided diffusion schemes are proposed to conduct image segmentation of road area,respectively.The experiments demonstrate that these training-free approaches detect the road region fast,accurately and robustly,and achieve favorable results on the KITTI benchmark.4)Aiming at the problems that the current convolutional neural network is highly dependent on offline training and lacks generalization for unknown scenes,a strategy using the sliding-window-based scanning method on LiDAR imagery and the backgroundattention-based network is proposed for off-road free space detection through semi-supervised online training.First,the original 3D point cloud is re-organized to an ordered LiDAR imagery.Then,a sliding-window-based method is proposed to detect the free space in the LiDAR imagery.Next,the labeled point cloud is projected onto the RGB image space,and Delaunay triangulation is used to obtaining dense road and obstacle regions.At the same time,a sky region filling algorithm is also designed to generate the mask of sky region.After building the foreground and background mask,a background-attention based SwiftNet is proposed to improve segmentation performance for off-road free space.Meanwhile,online training based on semi-supervised learning is utilized to update the parameters of background-attention based network frame by frame,which makes the network generate complete free-space prediction results and improves generalization performance.The experiments demonstrate the effectiveness of this online training strategy. |
