Research On Distributed Reinforcement Learning Theory And Its Applications In Multi-robot Systems

Reinforcement Learning(RL) is a newly arising Artificial Intelligence(AI) means. Under the condition that the secret of brain remains concealed and the essence of intelligence remains unexplainable, traditional logic-based AI means have many shortages in defining the structures and characteristics of AI systems by observation and hypotheses. Therefore, people turn to the research field that AI systems learn to achieve intelligence gradually, RL is such a kind of machine learning means. RL devises the map policy from states to actions by "trial-error" principle and learns to react under all states, so that the adaptability and robustness of AI systems could be improved.A robot simulates a human being, while a multi-robot system simulates the human society. As learning, communication and collaboration are essential characteristics of human beings, so it is of great importance to perform researches on Distributed Reinforcement Learning(DRL) in multi-robot systems. Nevertheless, existing DRL algorithms suffer from the hardness of Structural Credit Assignment(SCA), the slowness of learning rate, and other problems, thus their application fields are strongly restricted. In this thesis, in-depth researches on DRL theory are made and the primary solutions of some existing problems are presented.The basic idea, architecture and main algorithms of RL are introduced systematically, and the specialties and application fields of RL algorithms are discussed.The architectures and main algorithms of DRL are researched, and the architectures of DRL are classified into four frameworks. The four frameworks are introduced and formalized, their specialties, their application fields and the relationship between their main components are researched.The SCA algorithms in DRL systems are researched. Considering the instance that the only two existing SCA means can not embody the contribution of agents, a non-linear programming SCA scheme based on comparison between agents' actions is presented. After that, a feasible SCA algorithm is acquired by decompose the calculation course, which is subsequently devised into an incremental SCA algorithm to meet the fact that RL is a kind of incremental learning algorithm.The information sharing means in Reinforcement Learning Individually(RLI) systems are discussed, and the specialties and application fields of some main information sharing means are compared. A colored-orbit state segmentation algorithm and an element-importance state segmentation algorithm are presented, which can compress the scale of state space and accelerate the convergence of RLI algorithms.Considering the instance that the learning space of a Reinforcement Learning in Groups(RLG) system grows exponentially to the numbers of agents, a prediction-based RLG algorithm is presented. The new algorithm can cut down the redundant state information, so that the composition intensity of learning space is decreased and the convergence of the learning course is accelerated.Taking formation problem of multiple underwater vehicles as an example, the application model of DRL in multi-robot systems is illustrated. Collision-avoidance learning and formation learning are implemented in the simulation, and the simulation shows that the vehicles can learn to form a regular shape and reform the shape rapidly when they overpass obstacles or the shape is disturbed.