Classification And Segmentation Of Fruit Point Cloud Based On Deep Learning

Fruit classification and segmentation is the basis of fruit location,automatic picking and yield estimation.With the continuous progress of 3D point cloud technology,classification and segmentation technology based on deep learning has made great progress in autonomous driving,smart home,face recognition and other fields,but it still faces great challenges in the field fruit point cloud task.In this paper,based on fruit point cloud data,a deep learning fruit classification and segmentation framework based on 3D semantics is constructed,which is conducive to improving the efficiency and accuracy of fruit recognition and providing practical technical support for other basic field work,such as real classification and efficient management.The main contents and research results are as follows:(1)A set of fruit point cloud data set is constructed.In the traditional production process of 3D point cloud data set,the cost and difficulty of obtaining point cloud by laser scanning are high due to the existence of blade occlusion,swaying and uneven light in the real scene.In order to solve this problem,Kinect v2 was used in this paper to take photos of fruit scenes in the field,and color 1080 p photos were obtained by Kinect v2.Moreover,Colmap was used to complete sparse reconstruction based on SFM algorithm,SIFT operator and nearest neighbor algorithm.Open MVS was used to generate a dense point cloud model of fruits in the field.A series of fruit point cloud data were preprocessed by voxel sampling.(2)Completed the fruit point cloud classification task combining Spatial Transformer Network method with Point Net point cloud classification network.Point Net is a deep neural network that works directly on point clouds.Due to the irregularly of fruit point cloud and the great influence of light,Point Net cannot flexibly learn local context,and the classification results in the pre-experiment of fruit point cloud classification are not ideal.In this paper,Spatial Transformer Network is embedded into Point Net to realize the improvement of the network.By applying spatial transformation to the input point cloud,different local neighborhood features are obtained,which is convenient for the network to flexibly learn the features of fruit point cloud.The comparative experimental results show that the method proposed in this paper has significantly improved the classification effect on fruit point cloud compared with other networks.(3)A fruit point cloud segmentation network based on RNN is designed.Classical segmentation networks RSNet and G+RCU based on RNN can extract local information through region division,but still need huge computational cost.In order to solve these problems,this paper proposes a new fruit point cloud segmentation network,which uses point cloud segmentation and RNN to effectively strengthen the connection of local features and improve the segmentation accuracy.Local context can be quickly extracted by dividing the point cloud into blocks of the same size and suitably overlapping.At the same time,bidirectional GRU is introduced to explore the correlation between adjacent blocks,so as to effectively aggregate block features.Comparative experiments show that the proposed method has higher accuracy in fruit point cloud segmentation data sets.