Apple Seedling Grading Method Based On RGB-D Information And Instance Segmentation

Quantities of apple seedlings need to replenish dead seedlings and rebuild new orchards every year in China,where accurate and fast seedling grading to ensure their quality before planting has become a crucial problem.Currently,it mainly relies on manual measurement of morphological parameters for apple seedling grading,which is time-consuming and labor-intensive.At present,although some researchers and companies have developed grading systems or sorting equipment of seedling or rootstock,they are only graded according to a small number of indicators,which cannot comprehensively evaluate seedlings.Therefore,this study develops apple seedling grading algorithm based on RGB-D(Red,Green,Blue-Deep)information with instance segmentation to achieve accurate grading of seedlings according to the classification standard.The main research content and conclusions are as follows:(1)Standard-setting of seedling grading and dataset construction.Firstly,according to the development of apple seedlings in China,apple nursery plant with self-rooting dwarfstock were determined as the research objects.After reviewing domestic and international standards,grading standards suitable of the apple nursery plant with self-rooting dwarfstock were selecteed.Then,Azure Kinect camera was used to capture images of apple seedlings,including color and depth images,and 3D point clouds,which were alignmented.Each morphological parameter of the seedlings was measured manually to test the effectiveness of the subsequent algorithm.According to the grading standard,root,mainroot,rootstock,graft union,and scion were labeled.For thin and long scions,three different labeling strategies were used to label scion,namely whole labeling,segmental labeling,and segmental-end-merge labeling.Finally,dataset was augmentated and produced according to the network requirements.(2)Segmentation method of apple seedlings based on Blend Mask and different labeling strategies.After analyzing and comparing the accuracy of segmentation networks,Blend Mask was selected to segment the high and low images of apple seedlings and to select the appropriate labeling strategy model.The results showed that the m AP(mean average precision)of the segmental labeling model(91.1%)were higher than those of the whole labeling model(84.1%)in the high images.And the AP of scion was increased by 34.6%.Since the segmental labeling model and segmental-end-merge labeling strategy were very similar,the m AP of these two strategies were the same.However,the m IOU(mean intersection over union)of the segmental labeling model(79.3%)was higher.Therefore,the segmental labeling model was chosen to segment apple seedlings in this study.In the low images,the m AP~s reached 94.5%using the same segmental labeling model.It cost about 285 ms by Blend Mask to segment an image with a resolution of 3840×2160.The segmentation precision and speed based on Blend Mask meet requirements of apple seedling grading.(3)Morphological parameter algorithm of seedlings based on segmentation results.According to the alignment relationship of color image,depth image and point cloud,the mapping method between color image and point cloud was studied.Combining the segmentation results by Blend Mask and point cloud,the calculation methods of seedling height,rootstock length and thickness,main root length and seedling diameter were designed based on feature endpoint extraction.And the methods of angle calculation based on linear fitting and statistics of the number of full buds based on linear regression model were developed.The mean(percentage)errors of seedling height,seedling thickness,rootstock length,main root length,main root thickness,angle,and number of full buds obtained by algorithms were 25.18 mm(1.53%),1.10mm(9.19%),5.89 mm(4.67%),6.16 mm(3.58%),1.67 mm(9.73%),1.29°(10.95%)and 1.54(10.99%).The correct grading rate reached 92.0%with a speed of about 10 s,which can use into apple seedling grading.(4)Design of seedling grading software and dynamic grading test.In order to simulate the task of assembly line seedling sorting in factory mode,dynamic grading test was designed.The results showed that mean absolute error of each morphological parameter of seedlings increased with increasing speed,while the grading accuracy decreased.In addition,20 cm/s was selected as the dynamic grading speed of seedlings,and the errors of main root length,main root thickness,seedling thickness,rootstock length,and angle were 1.51 mm,11.02 mm,1.41 mm,12.78 mm,and 2.02°at this speed,and the correct grading rate was 93.1%.To facilitate the observation results,apple seedling grading software was developed using Python and Py QT5,including image acquisition,seedling segmentation,morphological parameter calculation,and grading result display module.In addition,software could save the segmentation results,morphological parameters and grading results of each seedling to view and count the grading results of the whole batch of seedlings.In summary,this paper proposes a grading method based on RGB-D information and Blend Mask for apple seedlings to address the current problems,such as time-consuming and laborious manual grading.Among them,the proposed Blend Mask network model based on segmental labeling strategy can achieve accurate and fast segmentation.The algorithm designed by combining segmentation results and point clouds can achieve accurate acquisition and grading of morphological parameters on each tree of seedlings.To simulate the task of assembly-line seedling sorting in factory mode,dynamic grading tests were designed to select the appropriate dynamic grading speed of seedlings.The developed apple seedling grading software can observe the grading results.This study provides new ideas and explores new paths for factory grading and sorting of apple seedlings and further promote the industrialization,automation and intelligence of apple seedlings.