Research On Adaptive Dynamic Programming Based Nonlinear Control Theory And Optimization Methods

With the rapid development of information science and engineering technology,in-dustrial control systems become more large-scale,and contain complex nonlinear struc-tures and uncertainties.How to guarantee the system security and reduce costs is the core topic of modern control systems.Nonlinear optimal control theory is the key to solving the aforementioned issues.Therefore,it has received much attention.Traditional dy-namic programming methods generally suffer from the curse of dimensionality.In order to overcome this bottleneck,a novel approach called adaptive dynamic programming is proposed.Adaptive dynamic programming integrates reinforcement learning,adaptive control and optimal control theory,and utilizes neural networks for its implementation.Although adaptive dynamic programming is regarded as one of the most effective ways to solve the optimal control problem of nonlinear systems,there are still many open prob-lems.Based on adaptive dynamic programming,we further study the problems of non-linear robust control,H? control with constrained inputs,discrete-time non-zero-sum games and cooperative optimal control.The main works and contributions are given as below:(1)For the nonlinear systems with actuator faults and mismatched disturbances,the robust control issue of uncertain systems is converted into the optimal control prob-lem of auxiliary systems.Using adaptive dynamic programming methods to solve the corresponding Hamilton-Jacobi-Bellman equations yields the forms of optimal control.For different types of faults and disturbances,we design associated ro-bust control polices based on the obtained optimal control strategies.The selection of control parameters is derived through the Lyapunov stability theory.Based on the idea of the aforementioned design procedure,we further study the nonlinear systems with both actuator attacks and mismatched disturbances,and convert the robust control issue into the zero-sum game.Policy iteration and value iteration al-gorithms are introduced to solve the Hamilton-Jacobi-Isaacs equations.Finally,the robust controller is designed according to the zero-sum-game-based optimal con-trol form.We utilize the Lyapunov theory to provide the stability analysis and the neural network approximation error analysis.(2)For the uncertain multi-controller systems,the associated robust control issue is converted into the multi-player game problem.In order to solve a set of coupled Hamilton-Jacobi equations,a data-driven adaptive dynamic programming method is proposed according to the model-based iterative algorithm.These two algorithms are proved to be equivalent,which implies the algorithm convergence.Take the dual-controller system as an example.In light of the optimal control strategies.we design two robust control schemes for different types of actuator uncertainties,and provide their stability analysis results.From the stability analysis results,the selection condition of robust control parameters can be derived.To implement the proposed data-driven adaptive dynamic programming method,we construct a critic network and provide the corresponding convergence proof.(3)For the nonlinear systems with completely unknown dynamics and constrained con-trol inputs,we investigate the H? control problem by utilizing adaptive dynamic programming methods.It is known that the H? control problem can be viewed as the two-player zero-sum game issue.In order to solve the Hamilton-Jacobi-Isaacs equation,we propose a model-based simultaneous policy update algorithm,and provide its convergence proof.Then,based on this model-based method,we de-velop a novel data-driven model-free algorithm,which only requires the real system sampling data instead of accurate system models.To implement this model-free al-gorithm,we construct a critic network,an actor network and a disturbance network to approximate the iterative performance index function,control policy and distur-bance policy,respectively.The proposed scheme provides a unified and feasible theoretical framework to handle a series of optimal control problems.By simplify-ing or modifying the performance index function,this scheme can be extended to solve the problems of general optimal control,zero-sum game and tracking control.(4)For discrete-time multi-controller systems,we first formulate the multi-player non-zero-sum game,and then derive the forms of optimal control policies which satisfy the Nash equilibrium.In order to solve the coupled Bellman equations,a policy iteration algorithm for multi-player games is proposed and implemented by three-layer back propagation neural networks.Next,based on the critic-actor framework,we design a novel online tuning algorithm and prove the stability analysis result to be uniformly ultimately bounded.Finally,to guarantee the sufficient learning of neural network weights between each two layers,we introduce an action-dependent heuristic dynamic programming based optimization method.(5)For discrete-time multi-controller systems with control constraints,we further study the cooperative optimal control issues.We first formulate the cooperative optimal control problem,and derive the forms of cooperative optimal control policies along with the discrete-time Hamilton-Jacobi-Bellman equation.To solve this equation,a policy iteration based adaptive dynamic programming algorithm is proposed.Neu-ral networks are utilized to implement the algorithm offline.Considering the case with control constraints,we further investigate the constrained cooperative control problem,and present the convergence proof of the associated policy iteration algo-rithm.Subsequently,by constructing a critic network,constrained actor networks and unconstrained actor networks,we design an online simultaneous learning strat-egy.