Research On 3D Detection Based On Camera And Millimeter-Wave Radar Information Fusion

With the rapid development of information processing technology and hardware computing resources,various autonomous driving technologies have gradually become practical.Relying on environmental information obtained through perception technology,autonomous driving cars can make rapid and accurate driving decisions.As a basic perception technology,3D detection can provide the location and pose information in the real world.Current 3D detection methods heavily rely on cameras and LiDARs.However,these methods have some weaknesses.Cameras are not able to provide accurate depth information.While LiDARs have a short detection range and are susceptible to weather conditions.Millimeter-wave radar has a strong penetration ability and can provide the ranging capability,which can complement cameras.Based on the mentioned analysis,this thesis focuses on 3D detection methods based on camera and millimeter-wave radar information fusion,and the main innovation points and contributions involved are as follows:(1)This thesis first proposes a 3D detection method based on keypoint detection.This method uses a monocular image as input to identify the target in the image in 3D.The proposed method first uses the neural network to identify the center of the target,the vertexes of the bounding box,and the geometric information such as dimension and depth,and then estimates the 3D parameters through geometric constraints.The simple structure of the neural network reduces the need for direct regression from the network and exploits the advantage of the geometric features.Experimental results on the open-source dataset show that the proposed method achieves superior performance with a short running time.(2)Based on the previous research,a 3D detection method based on the feature-level fusion of millimeter-wave radar data and visual image is proposed.In this method,the millimeter-wave radar point cloud is first projected into the image plane and then merged into a two-dimensional feature of the same size as the image according to the statistical features,in preparation for subsequent feature extraction and fusion operations.Next,this thesis designs a feature extraction network for radar data and extracts features from the generated raw radar feature map.Finally,after fusing the deep radar features with the image features,3D detection is performed.Experimental results show that the feature-level fusion of the two sensors can make full use of the original information of the two types of data.The depth information provided by the radar data improves the 3D detection performance.(3)Based on the feature-fusion method,a 3D detection method based on the multiple levels fusion of millimeter-wave radar and camera information is further proposed.Based on the featurelevel information fusion method,the fusion of millimeter-wave radar information at the decision and data level is added to form a multi-level information fusion.The decision-level fusion uses radar points to adjust the confidence level of the estimated 3D bounding box.Data-level fusion uses radar points and images to generate radar attention matrices,and then uses radar attention matrices to locally enhance image features.Experimental results show that multi-level information fusion further improves the 3D detection performance based on feature-level fusion,while the computational complexity is increased.