Research On Apple Surface Defect Detection Method Based On Infrared And Visible Image Registration And Fusion

Globally,apples are the third most important fruit in terms of planted area and production after bananas and oranges.Apples are also one of the most important agricultural crops in China,with a wide planting area and a world-leading total production.However,compared to developed countries,China’s apple industry suffers from poor quality,low value-added products,small export volume and low product prices.The main reason for this is that the post-harvest processing of apples in China is not perfect and the products are put on the market immediately after picking,which leads to a lack of competitiveness in the market.Therefore,real-time classification and surface defect detection after apple picking are necessary.At present,computer vision technology is used at home and abroad to automatically detect apples mainly based on their size,shape,color and other characteristics,but because the grayscale characteristics of calyx,stem and apple surface defects are relatively similar,it is easier to interfere with the effect of apple surface defect detection.Therefore,in order to solve the current situation that the detection rate of mechanical damage represented by scratches is low,the infrared and visible image can be fused to make full use of the complementary advantages of the two type of image data to obtain more obvious defect characteristics on the apple surface.Thus,this paper investigates the application of infrared and visible image registration technique,infrared and visible image fusion technique and convolutional neural network model for apple surface defect detection with Yantai red Fuji as the research object,and the main contributions are as follows.(1)An infrared and visible apple image surface defect dataset is constructed.By analyzing the dual-light image acquisition task and designing the infrared and visible apple image acquisition device through which the dual-light apple images are acquired,a dual-light apple surface defect dataset is constructed.In addition,the dataset was divided into six categories: calyx,calyx+defect,stalk,stem+defect,defective fruit and intact fruit,considering the image acquisition perspective.(2)An infrared and visible apple image registration algorithm based on the active contour model is designed.In this algorithm,firstly,the Chan-vese model is improved to extract the active contour curves of apple parts in infrared and visible images;secondly,the edge feature points on the curves are resampled at equal distances to construct the feature point sets of infrared and visible apple images;finally,the optimal scale transformation factor and the optimal horizontal transformation factor are obtained by calculating the partial Hausdorff distance between the two feature point sets.Finally,the registered visible image is acquired based on the obtained affine transformation matrix,thus realizing the registration of the thermal infrared and visible images of apple.Through experiments on 15 pairs of samples,the precise matching rate and root mean square error of the algorithm are 96.5% and 5.1475,respectively,and the alignment success rate is 96% when the overall experiment is conducted on 50 pairs of samples.The results show that the algorithm has a good registration effect for infrared and visible apple images and can meet the needs of infrared and visible apple image fusion.(3)Proposed an algorithm for infrared and visible image fusion based on attention mechanism and Boosting-integrated.The algorithm constructs an end-to-end fusion model architecture,proposes a fusion network based on dual-attention mechanism and a training strategy based on Boosting-integrated.The two-stage training strategy is proposed in the training of the fusion network,which firstly uses a suitable loss function to train the dualattention mechanism fusion network,and secondly performs the Boosting-integrated training strategy for the fusion network at different time nodes.The experiments are conducted on the TNO public dataset,and the experimental results are EN: 6.9837;SD: 88.4389;MI: 13.9674;(6(6(6(6(6(6: 0.0677;VIF: 0.6941;MS-SSIM: 0.8707.The experiments are conducted on the infrared and visible apple image dataset B,and the experimental results are EN: 6.8579;SD: 138.4187;MI: 13.7158;abfN : 0.1007;VIF: 3.2228;MS-SSIM: 0.9734.The experimental results show that the algorithm has a good fusion effect for infrared and visible apple images,and can meet the demand of apple surface defect detection based on the fusion of infrared and visible images.(4)Research on apple surface defect detection method based on weight comparison transfer learning with Mobile Net V3.In this study,firstly,the pre-training weights of the Mobile Net V3 model are trained on the Image Net dataset,and secondly,the pre-training weight parameters of each network layer in the feature extraction part of Mobile Net V3 are compared with the default weight parameters of the model itself for transfer learning,and an adaptive transfer learning method is realized.Finally,the Mobile Net V3 model based on weight comparison transfer learning is constructed to achieve accurate detection of apple surface defects on the infrared and visible apple fused image dataset.The experimental results on the fused image dataset of infrared and visible apple surface defects established in this study show that the fused dataset has improved in the evaluation metrics of accuracy,precision,recall,and F1-score compared with the single-light dataset,and the model constructed in this study has better performance in the evaluation metrics of accuracy,precision,recall,F1-score,parameters,and time spent on single image detection compared with other classical convolutional neural network models.