Research On Targets Detection And Fruits Location Method Of Xiaomila Based On Deep Learning

Xiaomila(Capsicum frutescens L.)in Yunnan is a crop with unlimited growth and the same period of flowering and fruiting,traditional mechanical picking cannot meet its agronomic requirements.At present,manual picking is used,however,due to the decrease in agricultural population and the increase in labor costs,the cost of picking is increasing year by year,which restricts developments of xiaomila industry.Recently,the country’s strong supports for digital and smart agriculture industry have promoted the application of computer technology in agriculture,the improvements of equipment hardware promoted the continuous developments of image processing,and created the basic conditions for the use of deep learning in xiaomila fruits picking.In this paper,in order to solve the problems of low fruits detecting and location accuracy,caused by the small target,high occlusion rate,and similar color to leaves of xiaomila fruits in the natural environment,took the xiaomila in the green and mature period in the plantation area in Shupi Yi Township,Qiubei County,Wenshan City,Yunnan Province as the research object,based on deep learning technology to explore the target detection and fruit location methods of xiaomila.The main works of this paper were summarized as follows:(1)Agronomic investigation of xiaomila planting,analysis of fruit morphological characteristics and construction of dataset.The growth characteristics and planting patterns of xiaomila were investigated to determine the methods and time of image data acquisition of xiaomila.In order to avoid overfitting,according to the characteristics of the growth environment of the xiaomila fruits,data augmentation was performed on the collected datasets of xiaomila fruits to improve the generalization ability of models.Analyzed the shape and size distributions of fruits and demands for production to determine the labeling criteria for xiaomila fruits in the datasets.Manually annotated 12000 images and saved the dataset in the desired format for deep learning model training.(2)Research on objects detection of xiaomila fruits based on improved YOLOv5 s model.Aimed at solving the problems of missed detection of fruits with different shapes and sizes,severe occlusion and similar color to leaves,optimized and improved the model parameters and architecture,and proposed the YOLOv5s-CFL model for efficient and rapid target detection of xiaomila fruits in natural environment.In order to verify the improved effect of the model,the ablation study was used,and the performance of the improved model was compared with other deep learning models which widely used in agricultural engineering.The results showed that on the xiaomila image testset,the m AP(mean average precision)of the YOLOv5s-CFL model was 85.13%,and the model weight size was 13.8MB.The improved model enhanced the feature extraction and multi-scale detection effect,was more robust to the detection of xiaomila fruits under different lighting environments,and has achieved good detection results on the dataset.(3)Research on semantic segmentation of xiaomila fruit images based on improved Deep Labv3+ model.Directly using the semantic segmentation model to segment small targets in large field of images took a long time to process and the segmentation accuracy was not high.The YOLOv5s-CFL model was used to detect the xiaomila fruit images which didn’t overlap with the target detection dataset,and used the predicted frame coordinates of the accurately detected fruits to extract 6843 xiaomila fruit images,the images were manually labeled to construct a semantic segmentation dataset.Compared the segmentation results of PSPNet,U-Net,and Deep Labv3+ network models in complex field environments for xiaomila fruits,and the Deep Labv3+ network model based on four different feature extraction networks(Xception,Res Net-50,Res Net-101,Mobile Netv2).The results showed that the Deep Labv3+ network model based on the Mobile Netv2 feature extraction network achieved highest m Io U(mean intersection over union)of 81.33%,m PA(mean pixel accuracy)of 88.32%,and frames per second(FPS)of 62.78.While improving the segmentation performance,the model reduced the amount of computation and met the real-time performance for the semantic segmentation of the xiaomila fruits.(4)Research on the location method of picking points of xiaomila fruits in the field using Intel Real Sense D435 i depth camera.Combining results of target detection and semantic segmentation of xiaomila fruits,the two-dimensional coordinate of fruits provided by the segmented images and the distance information provided by the depth camera were extracted.Calibrated the depth camera to obtain the internal and external parameters and combined with the principle of spatial coordinate transformation,a method for picking location and pose estimation of xiaomila fruits in the field was proposed.A field experiment was designed to verify the accuracy and feasibility of the positioning method using Intel Real Sense D435 i depth camera,and the results showed that when the camera was 20~35cm away from the plants,for the xiaomila fruits within the field of view,the positioning errors were less than 4.1mm,3.7mm,8.2mm for Xaxis,Y-axis,and Z-axis,which met the needs of xiaomila robotic picking.