The Design And Implementation Of The Distributed Platform Focuse On Robot Simulation And Reinforcement Learning

With the development of its use in robot application scenario,Reinforcement Learning has showed a great potential;at the same time,the appearance of robot simulation framework has also provided a convenient method for training tasks based on Reinforcement Learning.However,most of Reinforcement Learning tasks are operated on single physical node,therefore,the performance of the physical node has become a bottleneck that limits the operation efficiency of the training task.It is urgent to find an efficient approach to solve the performance limitation of single physical node;meanwhile,when the training task is running on the distributed platform,the time consumption will increase sharply due to the delay of the physical network.Therefore,it is necessary to optimize the communication overhead.Based on the previous foreign and domestic research works,we design and implement a distributed training platform for robot simulation and Reinforcement Learning i.e.,Re-Ray,which can be served as a general platform for the robot simulation training tasks based on Reinforcement Learning algorithms.Then,we further propose ES-RPRS algorithm based on Evolution Strategies to provide a new thought for the optimization of the Reinforcement Learning algorithm in a distributed scenario.The major contents are summarized as follows:(1)A distributed training platform for robot simulation,i.e.,Re-Ray,is designed and implemented based on open-sourced distributed framework Ray,which is mostly used in the robot simulation training tasks based on the Reinforcement Learning algorithm.The key tasks of the Re-Ray platform include:firstly,designing the architecture of the proposed Re-Ray based on the distributed framework Ray;secondly,the design and implementation of the training platform Re-Ray are operated from four aspects including the application layer,the adaptive modification of the distributed framework Ray,the integration and modification of multiple simulation frameworks,and the container layer;finally,the performance of the proposed Re-Ray is verified through comparative experiments.Re-Ray can not only provide users with a richer variety and scenarios of simulated robots,but also make it more convenient to use for users.(2)To address the problem of network delay in distributed platform,ES-RPRS algorithm is proposed based on the Re-Ray.The proposed algorithm can not only improve the speed of training tasks,but also has low computational complexity and high parallelism.According to the requirements of the atomicity of the algorithm in distributed scenarios and the parallelism of the Evolution Strategies,we cancel the perturbations,which are in the exchange content between the working nodes.In turn,perturbations are generated by working nodes with the help of shared random seeds.Without affecting the training results,the proposed ES-RPRS accelerates the training procedure by reducing network communication overhead.Finally,the performance of ES-RPRS in terms of hyperparameters search is tested and verified.