Optimal Consensus For Multi-Agent Systems Based On Reinforcement Learning

At present,with the rapid development of artificial intelligence technology,reinforcement learning has made great progress in the field of distributed control by combining it with optimal control.At present,the research on the optimal consensus of multi-agent systems based on reinforcement learning is mainly focused on first-order homogeneous systems,and only the consensus control based on cooperative interaction is discussed.In the cooperative control process of multi-agent system,because there may be multiple consensus states and the execution environment may be different,the multi-agent system needs to group the system topology in order to achieve multiple consensus control,that is,the group consensus problem.Based on the shortcomings of the above research work,the paper discusses the group consensus of heterogeneous multi-agent systems and the complex relationship between agents based on the adaptive dynamic programming method.The main content of this paper is summarized as follows:(1)The problem of optimal consensus control of the second-order discrete-time multi-agent system composed of leader-follower agents is explored.Combining the optimal consensus problem with the characteristics of the second-order discrete system based on cooperative relations,the necessary conditions and strategies for solving the system's optimal consensus problem are obtained.The results show that the online strategy iteration algorithm not only stabilizes the distributed second-order dynamic system,but also the system can achieve optimal consensus.Simulation experiments based on the actor-critic neural network verify the correctness of the theoretical results.(2)Assuming that the dynamics of the follower agent and the leader agent are completely unknown,the problem of optimal consensus control with an unknown second-order dynamic system is studied.As a consensus problem based on feedback control,the second-order discrete-time multi-agent system model based on directed topology is constructed as an optimal tracking control problem through an online deep reinforcement learning method.The paper proves the optimality of the value function of the agent and the stability of the consensus error system based on the second-order system including unknown dynamics,that is,the final position and velocity state of all followers can be synchronized with the states of the leader.The experiment based on the model-free actor-critic neural network verifies the correctness of the theoretical results.(3)Based on a discrete-time heterogeneous multi-agent system,the optimal group consensus control problem is studied.The system in question not only includes completely unknown agents with first-order and second-order dynamics,but also includes the strength of cooperation-competition interaction.In the optimal consensus control problem of multi-agent systems,the Hamilton-Jacobi-Bellman equation is difficult to solve.In addition,it is difficult to determine the exact model of the system in the real world.Therefore,based on the actor-critic neural network,a model-free strategy iterative algorithm is used to solve the above two problems.Then,the stability of the global tracking error system is analyzed,and it is derived that the local tracking error and the estimated weight of the actor-critic neural network are uniformly ultimately bounded.Finally,three experiments verify the correctness of the theoretical results.