| With the seriously increasing of aging in rural areas,the existing citrus production model can no longer meet the growing production needs.Therefore,the research on automatic harvesting robots is of great significance to solve the labor shortage and promote the development of the fruit industry.At present,the vision system of the harvesting robot is facing many problems such as unsatisfactory fruit detection accuracy in complex environments,lack of research on fruit pose estimation,and poor robustness of branch keypoint detection models.To solve the above problems,this thesis uses deep learning technology to study citrus in natural orchards and conducts research on key technologies such as fruit detection,fruit position estimation and branch keypoint detection to provide technical support for the vision system of citrus automatic picking equipment.The major contents and results are as follows:（1）An on-tree citrus detection method based on a noise-tolerant RGB-D feature fusion network is proposed.To compensate for the shortage of citrus detection based on RGB images,a noise-tolerant RGB-D feature fusion network（NT-FFN）is proposed to eliminate the noise interference from depth holes while fusing the complementary RGB-D information.First,features are extracted from the RGB image and the depth image.Then,multi-modal features are fused at different scales.By adaptively focusing on the key regions in both spatial and channel dimensions of different modalities,the complementary information is learned and the noise from the depth image is suppressed.Finally,multi-scale RGB features and fused features are used to guide the network learning,which enhances the detection ability of objects with different sizes and further improves the noise immunity of the network.The experimental results show that the NT-FFN achieves a precision of 92.6%,a recall of 94.1%,and an AP50of 95.4%,which outperform the single-modal citrus detection and other RGB-D fusion strategies.The detection speed is 39FPS,which meets the real-time requirements.The proposed method effectively utilizes multi-modal data,improves the detection accuracy of on-tree citrus,and can be widely used in other kinds of fruit detection tasks.（2）A real-time citrus pose estimation method based on a single RGB image is proposed.Due to the instability of point cloud data in complex orchard environments,this thesis proposes a method to estimate citrus pose only using a single RGB image.The position and pose of the citrus fruit in three-dimensional space are represented by the fruit navel point and the normal of the plane where the fruit navel point is located.First,a 2D image-based citrus pose annotation tool is developed and a citrus pose dataset is constructed.Then,a multi-task learning model named FPENet is proposed to simultaneously locate the fruit navel point and estimate the citrus pose.In addition,a hyperparameter is introduced in the loss function to achieve the simultaneous convergence of the multi-task.The experimental results show that the AP of the citrus fruit navel point detection is 89.3%,the mean error of citrus pose estimation is 10.71°,and the pose angle deviation of more than 80%fruit navel visible citrus is less than 11.25°.This method demonstrates the feasibility of using a single two-dimensional image to estimate three-dimensional fruit pose in non-structural orchards and lays a foundation for safe and accurate citrus picking.（3）A citrus bearing branch keypoint detection method based on a multi-level feature fusion network is proposed.To address the problems of tedious orchard management operations and the poor robustness of traditional branch keypoint detection algorithms,a deep learning based bearing branch keypoint detection model is proposed to assist branch pruning during fruit harvesting.The model uses a top-down keypoint detection framework.First,a candidate area is generated according to the fruit-growing position and the fruit stem keypoint detection.Second,a multi-level feature fusion network is proposed to detect keypoint within the candidate area.The network can learn the spatial and semantic information simultaneously,and retain the structural information of the bearing branch.On the citrus bearing branch keypoint dataset,the multi-level feature fusion network achieves an AP50of 92.2%for all keypoints,an AP50of 88.4%for cutting keypoint,a model size of 21M,and a computing cost of8.9GFLOPs,which outperform some state-of-the-art methods,such as Hourglass and Simple Baseline.The proposed method provides a basis for the efficient and accurate location of the bearing branch to be pruned.（4）Picking test is carried out by integrating the vision system with hardware devices.The picking test in a simulated orchard environment shows that the deviation of the fruit pose within 15°would not affect the grabbing of the end-effector,and the success rate of citrus picking in a simple scene exceeds90%.These demonstrate that the accuracy of vision algorithms could meet the requirements of practical applications and provides a basis for safe and reliable automated citrus picking operations. |
