Image Recognition And Pose Estimation Of Orchard Grapes Based On Binocular Stereo Vision

China is one of the largest producers of grapes in the world.Its picking tasks are mainly done manually,with high labor intensity and high cost.With the aging trend of the population becoming increasingly severe,mechanized and intelligent fruit picking has become a future development trend.Robust picking robot vision system is an important prerequisite for achieving lossless fruit picking.However,due to the complex unstructured orchard environment and the uncertainty of fruit morphology,the fruit recognition and pose estimation technology of picking robots have become technical difficulties in this field.To this end,this paper takes Rose Red grape as the research object,and conducts research on image recognition,point cloud segmentation and 3D pose estimation of mature grapes in natural scenes.The specific content is as follows.(1)Aiming at the problem of picking robots in the recognition of grape images in natural environments,a multi-scale feature extraction convolutional neural network YOLOV4-De R is proposed.First,the data set of grape images is expanded by the method of data enhancement.The YOLOV4-De R network is designed based on the structural principles of densely connected networks and aggregated residual transformation networks.The network promotes the fusion and repeated use of multiscale features,while reducing the amount of model parameters,and enhances the network’s ability to recognize small fruits and overlapping occluded fruits.Finally,through illumination comparison experiments and performance comparison experiments on the same test set with other recognition models,it is shown that the improved recognition network YOLOV4-De R has strong robustness and faster detection speed,providing high-precision real-time for grape picking robots location.(2)In order to obtain accurate and rich three-dimensional spatial information of grapes,a point cloud segmentation method combining instance segmentation and binocular stereo vision is proposed.First build a ZED binocular stereo vision system,analyze camera distortion,camera calibration,and epipolar line alignment algorithms and complete the corresponding experiments.Then design a robust instance segmentation network De-YOLACT++,and based on De-YOLACT++ instance model fusion semi-global block matching algorithm to achieve the point cloud segmentation of grapes.Finally,the experimental results of point cloud segmentation show that the designed point cloud segmentation algorithm has higher segmentation accuracy and faster segmentation speed compared with the traditional area growing method both indoors and outdoors,so that the grape picking robot can more effectively obtain the information of three-dimensional space of the fruit in real time.(3)Because the stalks of grapes in natural scenes are usually thin and easily obscured,it is impossible to directly determine the picking point.This paper proposes a method for estimating the pose of grapes based on local points.First,point cloud filtering is used to remove the fruit point cloud for filtering processing,and the grape point cloud is sliced longitudinally based on the straight-through filter,and the centroid point of each layer of the point cloud is calculated;secondly,a three-dimensional straight line simulation based on random consistency is adopted.It is legal to fit the central axis of all centroid points,and estimate the posture of the grapes with the axis direction vector;at the same time,use the cylindrical fitting method based on random sampling consistency to estimate the center of the fruit to realize the grape positioning.After many test results,the proposed grape pose estimation algorithm provides a new way for the picking robot to obtain the three-dimensional pose information of the grapes.(4)In order to verify the feasibility of the above algorithm,the above algorithm is integrated into the binocular stereo vision system,and multiple physical prototype tests are performed in an indoor environment.First,convert the posture of each bunch of grapes into a suitable posture for picking,and then associate the picking posture with the coordinate system of the picking robot base with the eye-to-hand calibration technology to realize the grape picking robot picking operation and further verify of effectiveness and feasibility of the algorithm.