Research On Detection And Localization Of Citrus In Natural Environment Based On Improved YOLOv7

Fruits are a major category of agricultural products.At present,fruit picking is still mainly manual,which is time-consuming and inefficient.The visual system of the picking robot is the key to obtain information of the fruit,and it plays a key role in completing the picking operation.Taking citrus as the research object,this paper studies the detection and localization of mature citrus in natural environment to improve the real-time and robustness of citrus picking robot.The main contents and conclusions of this study are as follows:(1)A dataset of mature citrus was constructed.Two devices,camera and mobile phone,were used to take pictures of mature citrus at different times and lighting angles in the orchard three times.The number of citrus pictures was augmented through a variety of image processing methods,and in view of the overlap and occlusion problems in citrus detection,representative samples were selected from the collected pictures to represent lightly and heavily occlusion test sets.Manually label all pictures,as to obtain a citrus dataset required for this study.(2)Construction and research of citrus detection model.To address the problems of low accuracy and slow detection speed during citrus detection in natural environment,this study proposed the LT-YOLOv7 model based on the YOLOv7 model.Firstly,the lightweight feature extraction network Rep VGG was used as the backbone to strengthen the citrus feature extraction capabilities in complex scenarios;secondly,deep separable convolution was used in the neck to effectively reduce the amount of parameters;then,the Efficient Channel Attention(ECA)was used to enhance the important multi-scale features obtained by the backbone,thereby maximizing the performance of the detection model;finally,the soft DIo U Non-Maximum Suppression(soft DIo U＿NMS)algorithm was used to optimize the screening of the bounding boxes,and further improve the ability to detect overlapped citrus.(3)Research on the localization of citrus fruits.Based on the binocular vision system and camera calibration,a binocular stereo vision system for citrus was established.The ZED binocular camera was calibrated by Zhang’s calibration method,Matlab was used to perform calibration experiments on the camera,and the internal and external parameters and correlation matrices of the binocular stereo camera were accurately obtained,and the calibration results were analyzed for errors.A citrus localization method based on binocular stereo vision was proposed to achieve citrus localization.The features were extracted through the Semi-Global Block Matching(SGBM)stereo matching algorithm,and the corresponding pixels of the pictures captured by the left and right cameras were matched to obtain the parallax value,and calculate the depth value of citrus through the formula.(4)The detection and localization methods of citrus were experimentally verified separately.In terms of citrus detection,by comparing with Faster R-CNN and YOLO series,the LT-YOLOv7 has the best performance.Its Average Precision(AP)value was 98.14%,the F1 value was 0.94,the detection speed of citrus on GPU can reach 208 frames/s and 4.8ms for a picture of 640×640 pixels,and LT-YOLOv7 occupied 32 MB of memory.Based on the binocular stereo vision,11 experiments were conducted and the results were recorded using the control variables method to locate the center point of the citrus.By comparing with the actual data measured by the Xiaomi laser rangefinder,the errors in the depth Z under normal light,front light and backlight conditions are-6～7 mm,-7～7 mm and-6～7 mm respectively,and the relative error is within 1.3%.In summary,it can be seen that the positioning method used in this study meets the picking requirements of the picking robot.