Construction Of Orchard Three-dimensional Point Cloud Semantic Map Based On Data Fusion

Agricultural robots are a new type of intelligent agricultural machinery with a wide range of applications in tillage,plant protection,harvesting,and weeding.Because China is a significant fruit-growing country,orchard robots and related core technologies are in high demand to meet the requirement for intelligent management of modern orchards.The threedimensional point cloud semantic map is a powerful method to improve the environment perception and scene understanding ability of orchard robots.This research takes the cherry orchard as the experimental scenario and proposes a construction method of a threedimensional point cloud semantic map.This method uses the result of image detection results and point cloud data fusion to annotate the semantic information for the point cloud.On the basis of obtaining the keyframe information,the semantically annotated point cloud data are spliced to build a 3D point cloud semantic map.To verify the above method,this research studies the joint calibration of sensors,trunk detection,and the construction of a point cloud semantic map.The main contents and conclusions of the research are as follows:(1)Joint calibration of sensors.To obtain the spatial mapping relationship between two sensors of the camera and Li DAR,the joint calibration experiment of the sensor was carried out,and the spatial mapping relationship of sensors is solved using the Pn P algorithm.The spatial mapping relationship of sensors is a prerequisite for the fusion of image detection results and point cloud data to complete the semantic annotation of the point cloud.(2)Trunk detection.To detect cherry trunk,several deep learning target detection methods are adopted and compared,and the optimal model of trunk detection is obtained.2000 images were collected and selected to construct a cherry orchard scene image dataset.Yolov3 was selected as the target detection algorithm to detect the trunk region of cherry trees,and the accuracy,recall,and average accuracy values were 93.44%,94.61%,and 93.79%,respectively,after model training.All the above three performance metrics outperformed the Faster R-CNN method and the SSD method.(3)Construction and evaluation of point cloud semantic maps.Two major issues must be addressed when creating a point cloud semantic map: semantic information labelling and point cloud semantic map splicing.The semantic information annotation of the point cloud is based on the spatial mapping relationship between the camera and Li DAR,and fuses the image detection results and point cloud data to give semantic information to 3D point cloud data.The splicing of the point cloud semantic map is based on the keyframe information on the point cloud map(based on the intermediate result of a Lego-Loam algorithm),and the keyframe point cloud data labelled with semantic information are further spliced into the 3D point cloud semantic map;for the test scenario,this research designs the accuracy evaluation method and index of the point cloud semantic map,and quantitatively evaluate the two key performance of point cloud stitching accuracy and point cloud semantic annotation accuracy.Among them,the two evaluation indexes of point cloud splicing accuracy,the average absolute error of distance and angle are 0.15 m and 2.53° respectively;The average values of translation error,rotation error,and mapping error are 0.58 cm,0.89°,and 0.42 pixel,respectively.The 3D point cloud semantic map constructed in this research contains both spatial location information of the scenario and semantic information of typical targets in the scenario.It is expected to provide key information for typical tasks such as automatic navigation control and precision spraying of orchard robots,which improves the environment perception and understanding capability of orchard robots.