Accelerating Reinforcement Learning Training In Robotic Tasks

As an important method in the field of machine learning,reinforcement learning has been widely used in robotics tasks.However,training via reinforcement learning method in robotics tasks is very time-consuming,mainly due to the complexity of the training environment for robotics tasks.In robotics research,to avoid damage to real robots,re-searchers usually use simulators to train policies.Training an effective policy model in a simulator faces two problems: 1)The policy model must be trained in a high-precision simulation environment to ensure that it is suitable for the real world.2)Even using ultra-real-time simulation training in the robot simulator still requires a long simulation time,and training in the simulation environment with higher precision is slower.In recent years,research on reinforcement learning training acceleration methods has been exten-sive,but most of the work cannot be effectively extended to the simulation environment commonly used in robotic tasks,and it is difficult to play a role in robotic task scenarios.In this thesis,the research on the methods for reinforcement learning training accel-eration is carried out in the background of robot tasks,starting from the precision char-acteristics of robot simulators and distributed computing technology,and design specific methods to improve the efficiency of reinforcement learning training.Specifically,the main work of this paper is divided into the following two parts:(1)Propose a reinforcement learning training acceleration method based on sampling in several different precision simulation environmentsIn this thesis,we analyze the relative characteristics of simulation precision and sim-ulation time in ultra-real-time simulation under robot simulator,and propose a method to accelerate reinforcement learning training by sampling in several simulation environ-ments with different precisions.In this method,a variety of simulation environments with different precisions are selected for sampling during training,and the advantages of more accurate training models in the high-precision simulation environment and faster training in the low-precision simulation environment are comprehensively utilized.Two different training modes are proposed in this thesis: 1)Sequential acceleration mode: the reinforcement learning agent samples in a low-precision simulation environment to learn a base model,and then samples in a higher-precision simulation environment to fine-tune the model.2)Joint acceleration mode: the reinforcement learning agent simultaneously samples in several simulation environments with different precisions during training.(2)Propose a distributed reinforcement learning framework that supports parallel sampling in different simulation environmentsThis work extends the reinforcement learning training acceleration method sampled in multiple simulation environments with different precisions to a distributed reinforce-ment learning framework.During the training process of the distributed reinforcement learning framework,the agent can only use the same environment for parallel sampling,and does not support sampling in multiple different environments at the same time.In this work,by analyzing the communication and computing methods of existing robot simulators and distributed reinforcement learning frameworks,a distributed reinforce-ment learning framework that can use robot simulators for parallel sampling training is designed.Based on the combination of robot simulator and distributed reinforcement learning framework,we abstractly encapsulate the sampling process of the distributed reinforcement learning framework to support parallel sampling in different simulation environments.The framework proposed in this work can be combined with the above training acceleration methods to further improve the training efficiency of reinforcement learning under robot simulators.In this thesis,two different robot task simulation scenarios are built,and the methods proposed in this thesis is experimentally verified from the aspects of training time and model performance.Experimental results show that our work can effectively improve the training efficiency of reinforcement learning in robotics tasks without losing model performance.