Research And Implementation Of Variable-Sized Product Packing Method With Robotic Arm Based On Heuristic Deep Reinforcement Learning

With the growth of e-commerce and advances in logistics automation,there is an increasing interest in developing intelligent robotic systems to replace repetitive,tedious and inefficient human labor in a warehouse.In particular,as a key component of warehouse automation,autonomous product packing based on robotic arms is an emerging research focus while also being a major challenge in the robotic community.Generally,in a product packing task,the robotic arm is asked to first pick an object from an unstructured pile of products and then place it in a packaging box to complete customer orders.Within such a task,two main difficulties are brought up in the picking and the placing steps,respectively.First,picking up products in unstructured scenes is not trivial,which could be more difficult when the product size is diverse and the scene includes occlusions.Second,placing products in confined spaces without collision and extrusion while making good use of the space is quite challenging.The existing studies are mainly focused on the picking part.After the object is captured,these methods put it wherever it wants.And some papers treat bin packing as a mathematical optimization in deep reinforcement learning,but the exploring space of bin packing is too large,the convolutional deep reinforcement learning is difficult to achieve a good optimization effect.In order to solve these difficulties,this paper proposes an intelligent packing algorithm based on heuristic deep reinforcement learning and a complete system,which can pick up objects in the disorderly pile of products with any posture and place them reasonably,so as to place more products in a compact way.For unknown object,the optimal position is difficult to learn and has bad convergence.For this problem,this paper improved the structure of deep reinforcement learning(DRL)for bin packing,by leveraging the imperfect experience of human packers.The creative combination of human experience and DRL,guided the agent to learn,but not limit its exploration,which help the agent learn more quickly and obtain more effective experience.Additionally,the invalid action mask was designed for removing invalid actions which greatly speeded up the learning process.To solve the problem of picking and placing in unstructured real scenes,this paper presented the PackerBot,a complete robotic pipeline for performing variable-sized product packing in unstructured scenes.We added a 6-DoF suction-based picking module and a product size estimation module,leading to a complete product packing system.And a transfer module was integrated with the system to transfer the optimal policy learned in simulation to real world.In this paper,a simulation platform is specially designed to train and test the proposed methods.In addition,the PackerBot system was demonstrated on the UR5 manipulator platform and compared with human intelligence.Extensive experimental results show that our method achieves state-of-the-art performance in both simulated and real-world tests.