Encoding Robot Topology Information For Deep Reinforcement Learning With Continuous Action Space

In the context of reinforcement learning,the training efficiency decreases exponentially with the size of state space,especially in the robot control problems with continuous action space.It is difficult to learn a real-time control policy for a real robot by reinforcement learning due to the high-dimensional continuous state space and action space.To solve the problem of low data efficiency in high-dimensional continuous search space,researchers have tried many methods,such as generating more good data to guide policy searching.However,to introduce useful prior knowledge is another way.But still,how to design state space representation which is easy to optimize and can effectively encode domain knowledge has remained an open problem.This paper presents a new state space representation framework by encoding topology information(such as the geometrical and kinematic relations among different parts of the robot)with relative geometrical models.The relative geometrical models of each joint of robots are constructed with the Lie group elements,and the topological information of robots is introduced into the state space as constraints explicitly.Under this constrained state space,the process of learning the policy by maximizing reward signal received through interaction with the environment is more efficient,and the validity of this state space construction method proposed in this paper is verified in three experimental scenarios.The main results of this paper are as follows:· Firstly,a state space representation method with topological structure constraints is proposed by using a special Euclidean group for manipulators.The results of the experiments showed that the training efficiency of the reinforcement learning algorithm is significantly improved under the state space representation constructed by our method.Besides,the method proposed in this paper is universal and can be applied to all kinds of tasks related to the robot's motion state.· Secondly,a state space representation method with topological constraints for a biped humanoid robot with a complex structure is proposed.Starting from the modeling of the complex mobile robot,the process of applying our method is demonstrated in detail.The method of supervised learning is used to verify that,with our state space representation the convergence speed of the neural network is faster under the same training data,which means the introduction of topology information is more conducive to the optimization of neural network.· Last but not least,the ablation study was used to test the validity of each component of the state space construction model proposed in this paper,and the experiment results are analyzed in detail.According to the analysis,the component optimization of the state space construction model was carried out,which simplifies the state space construction model with topology constraints proposed in this paper while maintaining the same effect.