Multi-Agent Data-Driven Control In Temporal-Spatial Games

With development of artificial intelligence,data-driven control has become one of the hottest issues in control theory,which aims to characterize system dynamics and construct control protocol from data.The independence to dynamics makes it widely used in intelligent control.Recently,intelligent control faces many challenges,such as distributed communication,observation loss and resource constraints,which leads to high demand on stability,performance optimality and predictability for closedloop systems.To improve data-driven control algorithm,this thesis aims to construct temporal-spatial multi-agent systems and games for characterizing dynamics and realizing the data-driven(predictive)control.The core contents of this thesis are as follow:1.The data-driven distributed control is studied based on static and extensive spatial multi-agent games.By analyzing the distributed dynamics characteristics of multi-agent and static games,spatial extensive games are proposed to ensure the consistency of Nash solutions between static and extensive games.The difference of computational ability among agents is considered with combining the static and extensive games with optimal control.Synchronous and asynchronous distributed policy gradient reinforcement learning algorithms are proposed to solve Nash solutions and distributed controller.These algorithms are realized by a distributed actor-critic network,where local experience-replay mechanism and asynchronous weight update rules are designed by the methods of weighted residuals.The implement guarantee the stability and performance optimality of the closed-loop system.2.The data-driven predictive control is studied based on temporal multi-agent and homogeneous temporal games.In game theory,individual decisions induced by asymmetric information are connected with global optimum utility through Nash equilibrium,that is,global optimization is equivalent to local ones.Similar to Nash equilibrium,time consistency presents the same characteristics in optimal control.Therefore,the dynamic phase space could be reconstructed by embedding sliding window in system time series to characterize sequential control on state evolution,and temporal multi-agent system is obtained from dynamic phase space.Taking dynamic correlation between control components and state evolution in sequence as the nodes or timeline of temporal multi-agent,the homogenous temporal games are constructed by allocating timelines the same performance index.Combining homogenous temporal games with optimal control indicates the learning model of predictive control with the multi-step input.Taking the state evolution,input sequence and delay cumulative reward as interactive data,a multi-step reinforcement learning algorithm is proposed to approximate predictive controller when system dynamics are unknown,and the data-driven predictive control is implemented via neural networks.3.The data-driven multi-objective predictive control is studied based on temporal multi-agent and heterogeneous temporal game.Different from single-objective optimal control,time consistency is weakly dominant in multi-objective optimization.Therefore,multi-objective predictive control problem has multiple solutions.On the basis of data-driven predictive control,the temporal multi-agent are allocated with heterogeneous performance indices,and the heterogeneous temporal games are constructed in prediction phase space.The heterogeneous temporal games is combined with multi-objective optimal control,and the state evolution,input sequence and delay cumulative reward under the heterogeneous indices are taken as the timeline interaction data.A heterogeneous reinforcement learning algorithm is proposed to approximate multi-objective predictive controller when system dynamics are unknown,and the datadriven multi-objective predictive control is implemented via neural networks.4.The differences between temporal and spatial multi-agent games are studied,and the data-driven predictive control with state observation loss is proposed.By analyzing dynamic constraints of temporal multi-agent and temporal games,the constraint relationships between temporal and spatial games of linear system are described via algebraic Riccati equation in phase space,and the dynamic consistency and time consistency in equilibrium of temporal and spatial games are guaranteed.The predictive control is remodeled as optimal control of multi-input systems in phase space,and data-driven predictive control with state observation loss is proposed from this optimal control.Though the state observation exists loss,the control guarantees the stability,state transition consistency and performance optimality of closed-loop system.5.The data-driven robust predictive control is studied based on temporal-spatial coupled multi-agent and coupled games.According to the independence of system inputs,the predictive phase space is divided into two independent parts.A twodimensional temporal multi-agent system is constructed,and temporal-spatial coupled games are designed to reformulate robust predictive control problem into solving Nash equilibrium.A binary phase reinforcement learning algorithm with distributed multistep inputs is designed to implement data-driven robust predictive control with unknown dynamic constraints.