Kiwifruit Detection And Localization Methods Based On Multi-source Information Fusion

The accuracy and robustness of the kiwifruit detection and localization in the field is further improved in order to overcome the difficulties of uneven exposure and occlusions between kiwifruit and other nontargets.Consumer-grade RGB-D cameras are used to obtain multi-source information of RGB,NIR,and depth images.The original convolutional neural network is improved according to characteristic of multi-source kiwifruit images.A kiwifruit detection and localization method based on multi-source information fusion was proposed to achieve fast and accurate kiwifruits detection and localization under the conditions mentioned above.The main contents are as following:(1)A dataset of multi-source aligned kiwifruit images is constructed.According to kiwifruit cultivation style,kiwifruit images are obtained from the bottom of the canopy.Two RGB-D cameras Kinect V2 and Real Sense D435 are secondary developed using SDK to obtain multi-source kiwifruit image.The images were taken in kiwifruit harvest time(September to November),different times of the all-day.The images of the original kiwifruit dataset are amplified by brightness transformation,contrast enhancement and reduction and etc.Manual labeling was used to annotate the ground truth data in RGB images and annotation information is mapped to the aligned infrared images.Finally,the obtained aligned kiwifruit dataset are renamed and divided into a training set and a test set in a 7: 3ratio.(2)Deep learning based kiwifruit detection methods were developed using multi-source information fusion.Faster R-CNN with VGG16 feature extraction network is chosen in the paper as the basic network.For accepting the multi-source aligned kiwifruit images,the original Faster R-CNN network is improved.One was VGG16 network that received RGB and NIR images simultaneously(Image-Fusion mode),the other was two VGG16 networks that received RGB and NIR images respectively,then being concatenated on the feature map(Feature-Fusion mode).Image Net pre-trained model parameters are used to initialize all networks,and back-propagation and stochastic gradient descent are used to train all networks in an end-to-end manner.(3)Experimental results and analysis of kiwifruit detection modes based on multi-source information fusion are studied.To evaluate the performance of kiwifruit detection modes based on multi-source information fusion,Image-Fusion mode,Feature-Fusion mode,RGB-Only mode,and NIR-Only mode are evaluated on the test set.Experimental results show that the AP value and total recognition rate of the Image-Fusion mode based on Kinect V2 are the highest,respectively 91.4% and 91.8%,which is improved by 2.1% and 2.2% compared to the RGB-Only mode.The detection speed is 0.131 s/image,which is similar to unfused modes.The Image-Fusion mode based on Real Sense D435 has the highest AP value and total recognition rate,respectively 91.4% and 91.9%,which is improved by 1.3% and 1.7% compared to the RGB-Only mode.The detection speed is 0.139s/image,which is similar to unfused modes.The overall detection performance are greatly improved by the fused mode,especially the recognition rate of adjacent and occluded fruits,and the detection speed is basically unchanged.(4)Kiwifruit localization method based on RGB-D camera is studied.The localization accuracy of Kinect V2 camera and Real Sense D435 camera at different distances are compared in the chapter.the smallest 1mm depth error can be obtained by Real Sense D435 camera at a distance of about 1m from the bottom of kiwifruit.Zhang Zhengyou’s algorithm is used to calibrate the Real Sense D435 camera to obtain accurate internal and external parameters.The center point of the detection rectangular in the aligned color image is obtained by Python programming.Combined with the depth value of the same position in the aligned depth image,the coordinate transformation method is used to obtain the 3D spatial position coordinates of the center point of kiwifruit.The experimental results show that the average errors of X-axis,Y-axis,and Z-axis are 4.1 mm,3.8 mm,3.1 mm respectively.In summary,in order to overcome the difficulties of uneven exposure and occlusions between kiwifruit and other nontargets,a kiwifruit detection and localization method based on multi-source information fusion is proposed.The Image-Fusion mode improves the overall detection performance and the recognition rate of kiwifruits that are adjacent and occluded,while the detection speed is basically consistent with the unfused mode.The error of the kiwifruit localization method in this paper is much smaller than the designed requirements of the kiwifruit robot arm.The research will further improve the speed and accuracy of kiwifruit detection and localization in the natural environment,thereby further promoting the mechanization and automation of the kiwifruit industry.