Driving Environment Perception Based On LiDAR Point Cloud And Visual Information Fusion

The development of autonomous driving technology has led to an increasing number of vehicles equipped with different types of sensors for the environment perception.Among these sensors,lidar and camera are the most eye-catching,but there is a great difference in data structure between the point cloud data of lidar and the visual information of camera,which brings some difficulties to the fusion.At present,the research of perception algorithm pays more attention to the single modal sensor,and the fusion algorithm has not achieved the desired effect.In the environment perception,3d object detection is a key task,which can provide an important premise for subsequent decision-making.In this thesis,3d object detection is carried out by integrating lidar and visual information,and multimodal data fusion method is explored.The main research contents are as follows:（1）The multimodal fusion theory is studied to solve the problem that the multimodal data augmentation method is limited.The lidar point cloud and the image are converted to a unified coordinate system,and global data augmentation and truth sampling data augmentation satisfying the consistency of multimodal data are adopted to improve the diversity of datasets,so that the model has stronger generalization ability and lays the foundation for the subsequent fusion.（2）Aiming at the fusion difficulties caused by the differences between lidar point cloud and visual information data structure,a multi-scale feature fusion strategy is proposed.Sparse point cloud is encoded as voxel to achieve regularization of data.The multi-scale attention module is added to the main network of point cloud feature extraction to adapt to the scale change of target.Image features corresponding to the voxel are determined by projection relation,and the feature fusion of point cloud and image is implemented.（3）To solve the problem of insufficient feature fusion between point cloud and lidar,a 3d object detection process based on feature-candidate fusion is proposed.Combining the lidar point cloud and image features,the 2d object detection results of images that are easy to train are used to assist 3d object detection.Based on the multi-scale feature fusion,a late fusion module is added to achieve multi-level fusion of lidar point cloud and visual information,which verifies the effectiveness of multimodal fusion.（4）A single stage and anchor-free feature fusion network is proposed to alleviate the problem that the efficiency of multimodal fusion could not meet the real-time requirements.On the premise of multimodal effectiveness,a point cloud pillars encoding feature extraction strategy is designed,and on the basis of acquiring point cloud and image feature,a single stage and anchor-free strategy is adopted to balance the efficiency and precision in the driving scenario.In the process of testing and verifying the dataset,the efficient 3d real-time multimodal fusion 3d object detection method achieves relatively stable and reliable test results.