| Multi-source sensor fusion perception and 3D reconstruction have always been research hotspots in the fields of unmanned vehicles,robots,intelligent devices,and VR/AR.These include multisensor fusion,high-precision map reconstruction,robust environment perception,and accurate path planning.With the development of multi-sensor fusion perception and 3D reconstruction related technologies,and the continuous deepening of my country’s exploration of the aerospace field,the research on the multi-sensor fusion perception and reconstruction system of unmanned vehicles in the aerospace field is becoming more and more important.However,there are still challenges in multisensor fusion perception and 3D reconstruction,such as limited types of sensors,low accuracy of spatiotemporal synchronization,lack of reconstruction information for complex and unknown terrains,non-robust of 3D perception for small objects,and data dependency.Therefore,this paper is based on the second-generation Mars rover and the terrain of the Mars field in the background of space exploration,combined with the variety of sensors on the rover and the terrain of the Mars field is complex and other characteristics,and the following work has been done to address the problems existing in the existing methods:(1)Multi-source sensor spatio-temporal consistent representation: Aiming at the problems of limited sensor types and low spatio-temporal synchronization accuracy in existing multi-sensor fusion algorithms,this paper proposes a spatio-temporal synchronization method for multi-source sensor fusion.The method includes time synchronization and space synchronization of more than five common sensors in related fields such as robots and intelligent driving cars.This paper starts from the software and hardware of various sensors at the same time.Firstly,the time synchronization of various sensors is carried out.Because there will be a certain time difference in the initial data collection of each sensor,it is necessary to keep all sensors synchronized with the system clock to ensure that sensor data can be obtained at the same time;Secondly,the spatial synchronization of multiple sensors is carried out.This part mainly performs internal and external reference calibration of various sensors to determine the parameters of the sensor itself and the spatial position relationship between the sensors.High-precision spatio-temporal synchronization results are also the guarantee for multi-sensor fusion perception and 3D reconstruction.(2)Multi-layer map construction based on multi-source information: Regarding the problem of poor perception ability for complex unknown terrain reconstruction in adverse environments such as night-time in existing map reconstruction algorithms,this paper designs a multi-source sensor information-based multi-layer map reconstruction method.The method first reconstructs the threedimensional map based on the multi-view MVE algorithm;Then,constructs a multi-layer map through the internal reference and relative position relationship of the thermal imager,multispectral camera and RGB camera obtained by multi-sensor space synchronization;Finally,a multi-source multi-layer point cloud map containing not only color but also thermal and spectral information is formed,which provides more accurate and comprehensive map information for tasks such as path planning and environmental perception.(3)3D environment perception based on binocular vision: To address the problems of non-robust3 D perception of small objects and data dependence in existing 3D environment perception algorithms,this paper proposes a 3D environment perception method based on 2D perception + 3D mapping using binocular stereo vision.Firstly,instance segmentation based on the SOLO algorithm is performed on the left image of the binocular camera to achieve object perception in 2D;Secondly,use the binocular stereo matching network BGNet based on deep learning to restore the disparity map in the coordinate system of the left camera,and convert the disparity map into a depth map and point cloud;Finally,2D objects are mapped to 3D point clouds,and constraints are applied to the 3D point cloud of the objects to obtain the final perception result.And this method not only allows for the perception of smaller objects but also does not require any 3D annotation.The experiments showed that using binocular stereovision and instance segmentation algorithms can effectively improve the robustness of 3D perception for small objects,and no 3D annotation is needed,making the perception results more accurate and reliable. |
