| With the rapid development of biotechnology and genetic technology,breeding is now getting more and more attention,while its subdiscipline,plant phenotyping is also getting more and more attention.Besides,as a requirement of precise agriculture,the phenotyping is even more necessary for the development of modern breeding technology and smart agriculture.As one of the most widely grown and accepted fruits,production methods of apple have changed significantly over the past half century,with apple seedlings changing from tall trees to dwarf rootstocks and from large crown thinning to dwarf rootstock dense planting with higher yields.Despite the changes in apple planting methods and seedling varieties,the phenotyping is still mainly fulfilled manually,which is time-consuming,expensive,and easily affected by different surveyors.In order to meet the needs of precise agriculture,a contact apple sizer was designed to fulfill the phenotyping of an individual apple tree with different accuracy in conjunction with a smartphone-based heterogeneous binocular vision apple tree phenotype measurement app.The following are the main research contents and conclusions of this paper:(1)Contact apple sizer based on angular displacement measurement.According to the research and analysis of traditional contact apple size measurement methods,the problems of traditional apple tree size measurement using vernier calipers were summarized.Besides,in order to solve the problems of small outer measuring surface of its outer measuring jaw,the standard of tightening while measurement,inconvenience of read and save the measurement results,difficulty of fulfill the measurement for same position on the fruit,and not easy to operate with one hand,the improvements such as increasing the outer measuring surface,using torsion springs and other elastic mechanical structure,saving and displaying the results on the Android smartphone were adopted.And the Hall effect angular displacement sensor were also adopted as the sensor of apple sizer.Finally,we used the ESP32 microcontroller with Bluetooth module as control module.And 3D printed components with some standard components were utilized to manufacture the apple sizer.Finally,the manufactured apple sizer was evaluated with the measurement results of vernier caliper,while the measurement results of vernier caliper were taken as the ground truth value.The apple sizer achieved 1.046% and1.963% of MAPE(Mean Absolute Percentage Error),0.823 mm,1.255 mm of MAE(Mean Absolute Error)for apple horizontal diameter and vertical diameter.(2)Phenotypes and image data of apple trees were collected,to further constructed the apple tree phenotype dataset.In addition to contact measurements of some apples on apple trees using apple size measurement devices,other phenotypes were estimated using noncontact methods to obtain more comprehensive information on apple tree phenotypes.At various times of the day,the height,ground diameter,and trunk diameter(or diameter at breast height,DBH)of 110 apple trees and 120 apples size information(including horizontal and vertical diameter)were measured using tools such as soft ruler,tape measure,escalator,tree measuring tape,and vernier caliper as the ground truth values of phenotyping of apple trees.Besides,a pair of heterogeneous binocular image of the measured apple tree was obtained using smartphone-based heterogeneous binocular camera.After that,300 apple tree images were also captured using different smartphones for training the neural network.Label Img image annotation tool was used to label the whole apple trees,apples,grafting joints in PASCAL VOC(Pattern Analysis,Statistical Modelling and Computational Learning Visual Object Classes)format dataset standard,while the trunk was labeled by a special subdivided label strategy.To ensure the diversity of the dataset and the robustness of the trained neural network for apple tree images with different environments and different illuminations,the dataset was augmented using brightness variation,histogram equalization,mirroring and random 30 degree rotation.Finally,the dataset is transformed into the format for YOLOv5 s network training,and the dataset registration and dataset partitioning are also completed.(3)YOLOv5s for detection of apple tree phenotypes.According to the needs of apple tree phenotype measurement APP deployment in this study,the YOLOv5 s network,which has the second least hyperparameters in the YOLOv5 series,was selected for detection of phenotypes.The apple tree phenotype dataset was applied to fulfil the transfer learning of pre-trained YOLOv5s model in the Py Torch framework for 400 Epochs(90,000 Iterations).The final trained YOLOv5 s model was evaluated in the Py Torch framework,whose average precision for apple fruit,grafting joints,trunk,apple tree is 95.31%,79.32%.84.29% and 76.53%,respectively.Afterwards,the model was pruned,int8 quantized using onnx-simplifier,onnxoptimizer and ncnnoptimizer to fit the deployment on Android smartphones.And the evaluation for the pruned NCNN YOLOv5 s models were fulfilled.Using the pruned NCNN YOLOv5 s model for inference on Android smartphone,and comparing with the test dataset,the final pruned NCNN YOLOv5 s model on Android devices for the four classes of apples,apple trees,trunks,and grafting joints reached average precision of 96.14%,80.63%,82.56%,79.17%,and mean average precision of 84.62% in the PASCAL VOC 2012 standard,which basically fit the minimal needs of detection of apple tree phenotype.(4)Apple tree phenotyping APP with smartphone-based heterogeneous binocular vision.The commonest used backward multi-camera specifications in the smartphone market are analyzed to illustrate the composition of smartphone-based heterogeneous binocular cameras through the imaging characteristics and parameters of different lenses.After that,a general method for recovering a pair of standard binocular image from a pair of heterogeneous binocular image,i.e.,virtual zoom,is proposed based on the imaging principle and hardware system of the smartphone cameras.Furthermore,the matching range of template matching is constrained using the principle of epipolar constraint of binocular vision.And then,the template matching using normalized correlation coefficient method is applied to obtain the parallax and depth of the apple tree phenotype detected by YOLOv5 s in the standard binocular image recovered from the heterogeneous binocular image.Finally,five phenotypes of apple trees were estimated using the pinhole imaging model of camera combined with the depth information of apple tree phenotypes,which reached MAPEs of 6.00%,11.20%,11.45%,12.96%,13.73%,and MAEs of 184.15 mm,6.78 mm,15.50 mm,8.80 mm,8.50 mm for the tree height,trunk diameter,ground diameter of 110 trees,and the horizontal,vertical diameters of 120 apples,respectively.In conclusion,we developed an easy-to-use,low-cost apple sizer and a nearly costless smartphone-based heterogeneous binocular vision apple tree phenotype estimation APP for phenotyping of single apple trees under dwarf rootstock dense planting pattern to address various problems in traditional phenotype measurement and existing phenotype measurement studies to achieve systematic phenotyping of apple tree. |
PDF Full Text Request