Research On Cross-domain Fruit Object Detection And Tracking Methods Towards Robotic Plucking

As one of the key points of the 14 th Five-Year Plan,agricultural intelligent robot is the key to solve the aging population and the shortage of agricultural labor force.However,the agricultural environment is an outdoor unstructured scene,which has many problems such as many changes in features and dense occlusion,which brings many challenges to object detection and multiple object tracking in robotic grasp.This dissertation aims to solve the above problems by using computer vision,deep learning and domain adaptation methods in robotic picking.1.Agricultural scene features change a lot,such as illumination changes and crop growth changes.This often leads to the object detection model based on deep learning,whose training set sample feature distribution is inconsistent with the test scene feature distribution,resulting in degradation of detection performance.Therefore,this dissertation uses the domain adaptive method in transfer learning to improve the cross-domain robustness of object detection.On the other hand,in the previous domain adaptive object detection work,the method based on image style conversion is used,but in the agricultural scene,the generated image of this method often confuses the semantic features of different types of objects,resulting in negative transfer.Therefore,this dissertation uses the cross-domain object detection paradigm based on adversarial learning,proposes an image-level adaptive module based on pixel-semantic features,and an instance-level feature adaptive module guided by visual similarity.A large number of experiments are carried out on the apple cross-season detection task and the famous tea all-weather detection task,and the average accuracy of the detector is improved by 55.9% and 13.5% respectively,and the performance exceeds the SOTA method for multiple cross-domain target detection.2.In the agricultural environment,the characteristics of many occlusions make it easy to produce a large number of trajectory switching in multi-target tracking,which leads to a significant reduction in tracking accuracy.Therefore,this dissertation proposes a multi-target tracking algorithm based on multi-feature cascade matching,which combines target detection,Kalman filter,NWD distance and VLAD image retrieval algorithm,in order to improve the performance of multi-target tracking algorithm in agricultural multi-occlusion scene.In this dissertation,fruit counting experiments are carried out in dense and sparse fruit scenes to evaluate the performance of the algorithm.In dense scenes,the estimated fruit yield is 310,and the real data is292;In the sparse scenario,the estimated value in this dissertation is 44,and the real data is 38.In both scenarios,compared with the SORT method,the trajectory switching is greatly reduced and the tracking accuracy is improved.3.According to the research content of target detection and target tracking,this paper combines key frame detection and multi-parallax 3-D localization,and proposes a real-time vision perception framework based on eye-in-hand monocular camera to obtain 3-D positions and dimensions of multiple picking targets for robotic picking.Meanwhile,this paper builds a hardware system for robotic picking,and uses JAKA robotic arm and three-finger dexterous hand as picking actuators to complete fruit picking experiments.After the real-time analysis,the frame rate of the framework is about 25 FPS,which has high real-time performance.The success rate of this vision frame for apple picking was 70% with 20 experiments.To sum up,this dissertation makes an in-depth study on domain adaptive object detection and multi-object tracking methods under robot picking tasks,aiming to solve the performance degradation problems caused by the characteristics of many changes in features and dense occlusion in agricultural scenes.These methods are tested in different scenes,all of which get high-performance improvement.And combined with the vision research content to complete the robot fruit picking.Meanwhile,this article focuses on apples as the main experimental object,running through cross domain detection,tracking,and harvesting tasks.