Study On Fruit Tree Crown Volume Measurement Method Based On Deep Learning

Volume measurement is widely used in fruit tree yield measurement,breeding,tree structure evaluation,irrigation,variable spraying and precision fertilization and other aspects of orchard fine management which are important basic data of fruit trees.However,the traditional manual volume measurement methods have some defects of high labor cost,long measurement time,and imprecise measurement.At present,the most popular methods are based on lidar and vision.Lidar can obtain a relatively accurate point cloud structure inside the fruit tree.However,point cloud data does not distinguish objects.In the complex actual orchard scene,it cannot accurately distinguish between target fruit trees and non-target fruit trees,target fruit trees and surrounding weeds,and lidar is expensive and complicated to use,which is not conducive to being widely used.To reduce the cost,this paper adopts a vision-based method to complete a simpler and more accurate volume measurement.The main research contents of this paper include:(1)Fruit tree image segmentation based on deep learning.To complete the volume measurement of fruit trees based on visual image information,this paper uses the deep learning semantic segmentation method to complete the pixel-level segmentation of fruit trees’ crowns and stocks,collects and produces a modern orchard semantic segmentation data set.The model PSPNet and the lightweight model Bi Se Net are selected and ameliorated based on actual orchard conditions.The improved PSPNet can achieve a realtime inference speed of 48.71 fps,with a speed-up ratio of 83.5%.The improved Bi Se Net can achieve 83.25% segmentation accuracy,compared with the former model,which is increased by 2%.Both of them achieved better comprehensive performance.(2)Embedded hardware platform development and system construction.To ensure the actual operation effect of the vision-based volume measurement system designed in this paper between the rows of orchards.This paper which is based on the built crawler autonomous driving platform deploys and exploits embedded devices NVIDIA TX2 and NVIDIA AGX XAVIER,builds a deep learning system environment and uses ZED binocular camera to complete the image acquisition,and the Intel Real Sense D415 infrared camera is used to complete the overall application implementation.(3)Optimize and accelerate the inference based on Tensor RT.To ensure that the running speed on the embedded device side meets the real-time inference speed requirements.This paper uses Tensor RT to accelerate the inference process,and adopts the method of the network layer and Tensor fusion,and weight,and activation accuracy calibration.Redeploy the inference process of the network model without reducing the segmentation accuracy.After acceleration,Bi Se Net can reach a maximum running speed of 228 fps on a personal PC,and a running speed of 43.3fps on Xavier,the embedded development platform,while the accuracy is reduced by only 0.003%.This has achieved the successful deployment of deep learning in embedded devices.(4)Volume measurement based on infrared and monocular vision.This paper designs a volume measurement method that combines infrared depth perception and deep learning semantic segmentation.The trained neural network model is used to segment fruit tree images,and accurate tree trunk pixel range information is obtained after postprocessing,then corresponding the depth image to the infrared sensor to get the depth value of the stock distance from the camera.Calibration of the monocular camera to obtain the conversion relationship between the real-world length and the image pixel width and height,and get the real size of the pixels in the real world.Design a pixel rectangle voxel volume measurement algorithm,calculate the d-value distance between the camera to the tree stock and the camera to the crown,and the true volume of the tree canopy is calculated by accumulating a series of voxels.(5)Volume measurement is based on vision only.By way of further cost-cutting,this paper designs a volume measurement method based on monocular vision only.First,a data set based on binocular depth estimation is created,build and train the network monodepth v2,to realize putting an image as input and the network can output the distance between the camera and target option.The trained network model has the best accuracy measurement effect in the range of 250 cm to 450 cm,and the estimated error is from 1% to 7%.This paper improves the three-dimensional geometric slicing method,and designs a section and reconstruction method based on the crown shape.Based on the segmentation results of the canopy pixels,one-pixel height is used as the segmentation scale in the longitudinal direction,and in each pixel row,a series of consecutive pixels is used as a calculation block to fit the three-dimensional geometric shape and using the camera’s internal parameters to obtain the true distance value,calculate the volume value of each pixel height solid geometry,combined with the multi-angle volume measurement algorithm and obtained the accurate and real tree crown volume value.(6)Comparison and analysis of the results of manual and visual measures.Using the manual volume measurement to get 10 target fruit trees’ volume in the actual orchard,using the improved slicing method and geometric geometry method and take the results as ground truth,and collect multi-angle infrared and RGB images at the same time.Using the two-volume measurement methods designed in this paper to complete the volume measurement.From the measurement results,and the average measurement error of the infrared-based pixel rectangle voxel method is 0.1695,the average relative error rate is7.62%,the average measurement error of the pure vision-based coronal slice reconstruction method is-0.133 and the average relative error rate is 6.419%.After testing,the results of the two designed volume measurement methods and the manual measurement results have a strong correlation,which means these two methods designed in this paper can achieve excellent measurement results and are more simple and faster for accurate orchard volume measurement to meet the actual operating needs of orchard volume measurement.