Kiwifruit Canopy Image Segmentation And Multi-classes Fruit Localization Methods Based On Deep Learning

Deep learning has shown high accuracy and fast speed on field kiwifruit image detection.However,all kiwifruits have been labelled and detected as only one class in most researches for robotic harvesting,where kiwifruits occluded by branches or wires have been detected as pickable targets.It may lead to damage to kiwifruit harvesting end-effector or robot when accessing those fruits.Therefore,a study on canopy image segmentation and multi-classes kiwifruit detection and localization based on deep learning is performed to achieve the goal mentioned above.The main research contents and conclusions are as follows:(1)Two datasets for multi-classes kiwifruit detection and canopy image segmentation were constructed.According to the scaffolding cultivation mode and growth characteristic of kiwifruit,images were obtained by placing a camera underneath canopy,with its central axis perpendicular to the canopy.Images were taken at different times of the day with varied lighting,and a LED illumination was used to ensure constant illumination at night.Manual labeling was used to create ground truth data for networks training and testing after determing the classes of two datasets.To improve the overall learning procedure and performance,two different data augmentation techniques were used to artificially enlarge the number of training set,by means of feeding models with varied data.Finally,two augmented datasets were organized in specific formats as input data for the networks.(2)Canopy image segmentation and wire reconstruction for kiwifruit robotic harvesting.The dataset was heavily imbalanced where the pixels of calyx,branch,and wire were much fewer than background pixels.For the imbalanced kiwifruit canopy image segmentation,it was proven that the uniform weights assignation method outperformed the median frequency weights.To develop a better segmentation model for kiwifruit orchard image,four backbones,i.e.,Xception-65,Xception-71,Res Net-50,and Res Net-101,were trained based on DeepLabV3+ in this study.Results showed that Res Net-101 achieved Io Us(intersection over union)of 0.686,0.709,and 0.424 for calyx,branch,and wire,respectively,and the highest m Io U(mean Io U)of 0.694.It took about 210.0 ms to process a resolution of 512 × 341 pixels image,which could be acceptable for the kiwifruit harvesting robot.The CDR(correct detection rate)achieved by the PPHT was 92.4%.It only took 6.40 ms to process a 512 × 341 pixels image on an Intel Xeon E5-1650 v4(3.60 GHz)six-core CPU,which can realize a realtime application.Canopy image segmentation can provide a basis for guiding the harvesting end-effector to pick kiwifruits safely,thus improving harvesting success rate.(3)Multi-classes kiwifruit detection based on YOLOv5 s was studied.In order to deal with the new problem that some fruits not pickable for robotic picking detected by deep learning,and considering the experimental hardware platform and model performance,YOLOv5 s was utilized to detect multi-classes kiwifruits in field.For one fruit predicted as bboxs with two classes,a method named "class priority" was proposed to delete the lower priority bbox.Experiments on 232 test images showed that YOLOv5 s achieved APs(mean Average Precision)of 99.0%,96.1%,94.7%,95.1% and 96.4% for NO,OB,OW,OL and OF,respectively,and m AP(mean AP)of 96.2%,model size of 14.5 MB,and average detection speed of 14.8 ms.YOLOv5 s showed robust performance on multi-classes kiwifruits under different lighting and exposure environments.The bbox revision algorithms for one fruit predicted as bboxs with two classes proposed can effectively delete the lower priority bbox.(4)Research on kiwifruit localization based on MYNT EYE D1000-50/Color.In order to ensure the safe picking of the kiwifruit harvesting robot in field and realize the information confirmation of obstacles that were not pickable for robotic harvesting,the output results of DeepLabV3+ and YOLOv5 s were fused.Zhang Zhengyou’s calibration algorithm was used to calibrate MYNT EYE D1000-50/Color to obtain accurate internal and external parameters.Python programming was used to obtain the depth value of the center point of fruit bbox,branch and wire.Then the 3D spatial position coordinates in world coordinate system was transformed and achieved average errors of 8.4 mm,9.3 mm,and 10.4 mm for X-axis,Y-axis,and Z-axis,respectively.Finally,according to the 3D spatial position coordinates of branches,wires and multi-classes kiwifruit in field,a safe picking strategy for kiwifruits robotic harvesting in field was forecasted.In summary,this study proposed a feasible solution to the problem of excessive detection of kiwifruits occluded by branches and wires,that is,canopy image segmentation and multiclasses kiwifruit detection and localization based on deep learning.Among them,the multiclasses kiwifruit detection model based on YOLOv5 s had high m AP,fast speed,and smaller model size,which could realize multi-classes kiwifruit detection under different lighting and exposure environments.New ideas was explored to realize faster and more accurate detection and localization of pickable kiwifruit for robotic harvesting,so as to further promote the industrialization,intelligence,safety and automation of kiwifruit.