Research Of Mango Detection And Segmentation Based On Adversarial Mask R-CNN

The task of image detection and segmentation enables the machine to understand and perceive the surrounding environment,determine the specific location of the fruit and the growth information of the fruit itself,which is an important prerequisite for the production work such as automatic application of drugs,spraying irrigation and picking.In view of some problems in mango detection and segmentation in the natural orchard scene,such as uneven illumination of mango skin,branches and leaves occlusion and fruit overlapping,this paper proposes the Mask R-CNN algorithm which integrates GAN and the improved Mask Scoring R-CNN algorithm that integrates Boxiou.Finally,mango detection and segmentation system is designed and developed to achieve high-precision detection and instance segmentation of mango in orchard scene.The main research work and innovation content of this paper are as follows:（1）The establishment of mango database.Deep learning models often need to be trained based on large-scale data.The number,quality and diversity of model training set will directly affect the performance of the final model.In this paper,mango image data are collected from the natural orchard scene,including a variety of different degrees of uneven skin light,branches and leaves occlusion and fruit overlap;mango species include Tainong one,Aomang and Coconut mango,so as to obtain a variety of mango data.In addition,we use Labelme annotation software to mark mango instance mask,and then divide all data into training set,verification set and test set randomly to build mango data set.Among them,the test set includes uneven illumination,branches and leaves occlusion,fruit overlapping,small,medium and large targets and other categories.In order to evaluate the advantages and disadvantages of this model comprehensively and accurately,the evaluation indexes of COCO data set and PR curve are used.（2）Research on mango detection and segmentation based on Adversarial Mask R-CNN.In this paper,based on the structure of Mask R-CNN algorithm,a new full convolution network is added to the mask branch as the multi-dimensional feature fusion discrimination network,and the original Mask R-CNN structure is used as the generation network,and the two networks are trained with the strategy of alternating confrontation,so as to form the Adversarial network.Among them,smooth₁+IOU loss designed in this paper is used to GAN loss.Smooth₁ loss is used to measure the difference between the two feature matrices,IOU loss is used to measure the similarity of the two binary masks,and the combination of them can significantly improve the model performance.Finally,the segmentation AP of the total test set is 85.1%,the segmentation AP of the uneven test set is87.9%,the segmentation AP of the branch and leaf occlusion test set is 86.3%,and the segmentation AP of the overlapping test set is 81.1%.In addition,GN+WS optimization method is added to further optimize the training process of the model.At the end of the experiment,several comparative experiments are designed,including the performance of the model in different test sets,which further demonstrates the advantages of the improved algorithm of the fusion Adversarial network.（3）Research on the segmentation algorithm of mango detection based on Box IOU Mask Scoring R-CNN.In this paper,a Box IOU parallel branch structure is added after the ROI Align layer,which is composed of three layers of full connection layer.The IOU scores is predicted for the detection boxes,and then the scores is weighted coupled with the classification confidence as the basis of Non-Maximum Suppression arrangement in the final test stage,so as to screen out more accurate detection boxes,so as to improve the performance of the model.The coupling factor of IOU score was 0.6 through the verification set experiment.Finally,the performance of the improved algorithm is verified on several different test sets,and compared with other algorithms to verify the effectiveness of the improved Box IOU algorithm.Finally,the detection AP of the model in the total test set was 82.6%,The detection AP of uneven light test set reached 85.5%,the detection AP of branch and leaf occlusion test set reached 83.5%,and the detection AP of fruit overlap test set is 79.1%.（4）The design and implementation of mango detection and segmentation system.In this paper,the pages and functions of mango detection and segmentation system are realized by high-performance notebook computers,high-definition dome network cameras and other hardware devices of HIKVISION,as well as by combining Microsoft Visual Studio 2013 and Pycharm.System functions mainly include data selection,model selection,task selection and result display.It can detect,classify and segment mango image or video frame sequence offline or online.The main interface can display the information of the selected model,task information,execution time,number of mango targets,etc.in real time.