Development Of Apple Target Location And Picking Device Based On Monocular Image

China is a major country for both apple production and consumption.As a highly seasonal and labor-intensive task,apple picking has a direct impact on subsequent production and processing.However,traditional manual harvesting requires a large amount of labor input and is inefficient.At present,apple-picking robots have insufficient fruit positioning accuracy,complex harvesting device structures,high production costs,and cannot meet market demands,severely restricting the development of the apple industry.With the development of artificial intelligence technology and the transformation of fruit tree planting patterns,it is of great significance to optimize the recognition and positioning algorithms and develop a simple,low-cost harvesting device that is easy to operate and maintain to meet market demands.This study focuses on the automation of apple harvesting in wall-shaped orchards and carries out research on monocular image apple target positioning and harvesting device development.The main research content and conclusions are as follows:(1)Technical scheme design and dataset preparation.Firstly,the environment and spatial distribution of apple fruit in wall-shaped orchards were introduced to determine the overall technical scheme of the system.An image acquisition platform was built to capture color images and depth maps under different lighting conditions in the orchard.The fruit targets were labeled using Label Img,and the apple dataset was obtained by data augmentation methods such as horizontal mirroring,color jitter,and rotation.(2)A single-image depth estimation model based on improved High-Resolution Network(HRNet).To obtain high-precision dense depth information in the orchard,a multi-branch parallel encoder network was constructed based on HRNet.The dense connection mechanism was introduced to enhance the continuity of feature transfer,and the Convolutional Block Attention Module(CBAM)was used to recalibrate the fused features to improve the performance of the encoder,strengthen the structural information of the feature maps,and achieve effective extraction of multi-scale features.The Stripe Refinement Module(SRM)was introduced into the decoder network to highlight the edge features and improve the edge blur,and the depth map was generated by upsampling.Through transfer learning method,the model was trained and parameter fine-tuned on the orchard depth dataset using pre-trained weights on public datasets.Experimental results showed that the improved HRNet network generated depth maps with clear contours and retained more detail information.Compared with advanced depth estimation networks such as Dense Depth Net and BSNet,it performed best in subjective evaluation and objective indicators.The mean relative error(MRE),root mean square error(RMS),logarithmic mean error,depth edge accuracy error,and edge integrity error on the orchard depth dataset were 0.123,0.547,0.051,3.90,and 10.59,respectively.The accuracy at different thresholds of 1.25,1.25~2,and 1.25~33 reached 0.850,0.975,and 0.993,respectively.(3)Design of apple object localization algorithm based on monocular images.Compared with the current mainstream YOLO series object detection networks,the Vari Focal Loss was used for classification loss by asymmetrically weighting positive and negative samples through different parameters,which increased the proportion of high-quality positive samples’influence on the model.The SIo U Loss was used for localization loss to guide the network for rapid optimization.YOLOX-s was selected as the apple object detection model,with the precision of 85.4%and the mean Average Precision of 91.6%.The detection time for a single image was 21.2 ms,which could effectively complete real-time detection tasks.The average of the pixel coordinates of the upper-left and lower-right corners of the detection rectangle was taken as the two-dimensional coordinates of the apple center of mass.Meanwhile,in the depth map,the average depth value of the apple target,including the center of mass and its neighboring eight pixels,was taken as the apple target depth value,achieving single-image apple target localization.Experimental results showed that compared with the three-dimensional coordinates obtained using calibration boxes and real depth information,the two-dimensional coordinate error of the fruit center of mass within 1.5 meters could be controlled below 2 pixels,and the depth error was below 1.6 cm.(4)Research and development of the apple picking device.In order to meet the system development requirements while minimizing production costs,a simpler gantry-type robotic arm was chosen to complete the development of the picking device.The key hardware components of the apple picking device were selected,and the hardware circuit and control system were designed.A supplementary gantry-type hand-eye calibration test was also designed to obtain the three-dimensional coordinates of the apple target in the picking device base coordinate system.Finally,a simulation test environment was built,and the number and position of the hanging fruits were changed to conduct tests of the apple picking device.The average picking time per apple was around 15 seconds,and the success rate was above 80%.