Research On Real-time Detection Method Of Fruit Surface Defects Based On Deep Learning

China is the largest fruit producer and consumer in the world,and its annual import and export of fruit ranks among the top in the world.According to the latest data statistics of the Food and Agriculture Organization of the United Nations statistical database and the National Bureau of Statistics,China’s apple and tomato production in the past decade have shown a general growth trend,and accounts for a high proportion of the world’s total output.The appearance of fruit is an important basis to measure the quality of fruit,which affects the purchase intention of consumers,and then affects the economic income of fruit farmers.Therefore,it is of great significance to realize the in-time detection and grading of fruit surface defects.Based on YOLOv4 model and BiSeNet V2 semantic segmentation model,this paper takes tomato and apple as the research objects,and studies the methods of real-time detection and grading of tomato and apple surface defects.The main research contents of this paper are as follows:(1)Real-time detection of tomato surface defects based on YOLOv4 pruning model.Firstly,the tomato images are collected by the image acquisition system to make a dataset,and the image data is preprocessed.Then,tomato surface defects are detected based on YOLOv3,YOLOv4 and YOLOv4-tiny target detection networks.According to the comparison of evaluation indicators,YOLOv4 is selected as the basic research network,and the advantages and disadvantages of online detection of tomato surface defects based on YOLOv4 model are analyzed.Aiming at the issue of large size and long inference time of YOLOv4 model,the methods of channel pruning and layer pruning are proposed,which compress YOLOv4 network from two aspects of width and depth,reduce the size of YOLOv4 network model,and improve the real-time detection speed,so as to meet the real-time requirements of tomato defect online detection.Compared with YOLOv4 model without pruning operation,YOLOv4 P model after channel pruning and layer pruning reduces its size by 232.40 MB,forward inference time by 10.11 ms,and m AP by 2.11 percentage points.The comparison results show that model pruning can improve the detection speed effectively while maintaining the accuracy of model detection.In practical applications,if the model is pruned,the detection results of the target area will have the problem of redundant prediction bounding boxes.To solve this problem,an NMS method based on L1 norm is proposed to remove redundant prediction bounding boxes from model detection results.Finally,the performance of YOLOv4 P pruning model was verified,and tomatoes were used for online detection.The defect precision is 97.8%,the recall is 94.7%,the overall detection accuracy is 96.4%,and the F1 is 0.96.The result shows that YOLOv4 P network model has great potential in the practical application of tomato surface defect online detection,and has been applied to the tomato quality detection and grading production line in Linhai,Zhejiang.(2)Apple surface defect detection and grading based on BiSeNet V2 semantic segmentation network.In order to achieve apple surface defect detection and apple grading accurately,it is necessary to calculate area and quantity of the defect.In this paper,the Unet,DAnet and BiSeNet V2 semantic segmentation networks are selected to carry out research,and the detection and segmentation of apple surface defects are completed respectively.The pixel accuracy,parameter number,inference time and model size of the models are compared.The BiSeNet V2 network model can achieve the optimal segmentation accuracy and real-time performance.Therefore,BiSeNet V2 network model is selected as the basic research network for apple surface defect segmentation.When the BiSeNet V2 network model is used to segment apple surface defects online,it is found that although the BiSeNet V2 network model has better accuracy and faster detection speed than the Unet and DAnet networks,there are still a few cases of missegmentation in practical applications.The fruit stem or calyx is misidentified as defects.In response to this issue,considering the higher detection accuracy of YOLOv4 P.Firstly,the YOLOv4 P network model is used to detect the surface defect area of apples,and then the BiSeNet V2 semantic segmentation model is used to segment the defect to obtain accurate defect quantity and defect area.(3)Construction of apple surface defect area mapping model.Due to the angle of view,the size of the apple surface defect area in the image varies with the location of the defect.To address this issue,The calculation method for defect area was proposed.By establishing a functional relationship between the true number of pixels in the defect area,the distance from the defect area to the center of the apple,and the radius of the apple,a mapping relationship model was constructed between the number of pixels corresponding to the defect area in the apple image and the number of pixels corresponding to the true defect area.Due to the varying size of the apple,the distance between the apple and the camera lens can affect the accuracy of apple defect area mapping.In order to determine the corresponding relationship between the number of pixels in the defect area of apples of different sizes and the actual defect area,a method is proposed to determine the mapping relationship between the actual defect area and the number of pixels in the defect area by experiments.The test results show that the precision and recall of the three grades of apples are 94.30% and 94.33%,respectively,above 90.63%.The accuracy is 92.42%,and the F1 is 94.31%.Among the three grades,the apple with the highest accuracy is Grade I(95.59%),and the apple with the lowest accuracy is Grade II(92.06%).The test results show that BiSeNet V2 network model combined with YOLOv4 P network model can achieve accurate detection and segmentation of apple surface defects,and the inference speed meets the requirements of actual production,providing technical support for real-time apple detection and grading.This article proposes a model pruning method based on YOLOv4 object detection and an apple surface defect segmentation method based on BiSeNet V2 semantic segmentation network to address the accuracy and real-time issues in tomato defect detection.The method meets the requirements for real-time detection and grading of apple surface defects and is applied to Kangdun Smart Agricultural Production Management Center in Feicheng,Shandong for on-site verification.