| Environmental perception is an important module in the intelligent driving system.Different sensors play different roles according to their characteristics.Object detection based on a single sensor in complex scenes has some limitations.Therefore,fusion of multi-sensor modal data is crucial.Millimeter-wave radar is widely used due to its low cost,strong antiinterference ability,range and speed measurement and other characteristics.Visual images provide rich texture information,which have great advantages in the field of object detection.Therefore,research based on radar and image fusion will help to make up for the shortcomings of single sensor detection.This paper conducts in-depth research on millimeter wave radar and image fusion methods,and builds fusion models at decision-level,feature-level,and data-level.Specifically,the main work and innovations include:(1)An object detection method based on radar and image fusion at decision-level has been proposed.A distance adaptive clustering method called KMNN-DBSCAN is proposed for radar data to cluster point cloud,and the target bounding box is determined according to the aspect ratio of common traffic targets.A MobileNetv3-YOLOv4 object detection model for image data is introduced.And then,combining the two detection results,a fusion method based on the Intersection-over-Union of candidate bounding box for decision-level association is proposed.Experimental results show that this method is more robust and guarantees the balance of speed and accuracy.(2)An object detection method called SPCRF-Net based on radar and image fusion at feature-level has been proposed.This method preprocesses radar raw data into fixed-size line segments and maps them to the image.And then,spatial pyramid pooling module is introduced to expand the receptive field.This method uses VGG16 as the backbone network for image feature extraction.In the fusion level,SE attention module is introduced to enhance the ability of high-level feature perception.And then we constructed a fusion structure(PFPN)to enhance feature extraction.Experimental results show that this method can effectively reduce the missed detection of targets and improve the performance of object detection.(3)An object detection method called PSAG-CenterFusion based on radar and image fusion at data-level has been proposed.This method uses CenterNet to extract features of the image to predict 2D bounding boxes and generate the corresponding frustum ROI.It converts radar point cloud into a fixed height column and associate the frustum ROI to generate a threechannel heat map.After the image feature map is fused with the radar heat map,the PSA module is introduced to extract spatial information of different scales.And then Generalized Focal Loss is introduced in the regression task to further balance the problem of sample imbalance.Experimental results show that this method can significantly reduce the scale and angle errors of the target,and can improve the average accuracy of multiple categories. |
