Adaptive-Dynamic-Programming-Based Optimal Control Method For Nonlinear Systems

Recently,the optimal control problem for nonlinear systems has been one of the hotspots.The key point of the optimal control is to solve the Hamilton-Jaccobi-Bellman(HJB)equation of a given nonlinear system.Considering that the calculus of variations is not suitable to resolve the optimal control problem for nonlinear systems with bounded control input and the dynamic programming owns the "cure of dimensionality",adaptive dynamic programming(ADP),which combines dynamic programming,reinforcement learning and neural networks,has been widely studied and applied.On one hand,the ADP obviates the "cure of dimensionality" of dynamic programming.On the other hand,based on the universal approximation property of neural networks,the ADP can not only be applied to solve optimal control problems for known nonlinear systems,but also for unknown systems.Moreover,with developments of communication techniques,networked control systems have been widely adopted,and how to improve the utilization efficiency of network resources has been a concern to people.Different from the traditional control scheme,which transmits and updates the information periodically,the event-triggered control scheme has gained more and more attention,and the optimal event-triggered control problem has been widely proposed.Thus,in this thesis,the following optimal control problems are studied based on the ADP method:(1)For unknown continuous-time nonlinear systems with constrained input,a policy iterationbased adaptive dynamic programming algorithm is proposed to design the optimal control law.Generally,when system dynamics are unknown,model networks are usually applied to approximate unknown system models,and an optimal control law is designed based on the obtained model network.In this chapter,firstly,the quasi-model,critic and actor networks are proposed simultaneously to approximate the unknown function constituted by the input drift function and the partial function of the cost function with respect to the state,the cost function and the control law,respectively.It obviates the necessity of separately learning and training the model network in advance by introducing the quasi-model network.During the iteration process,weights of the quasi-model and critic networks are updated according to the least sum of square error,which makes the proposed algorithm efficient and computationally tractable.The optimality and convergence properties of the proposed method are proved based on the Lyapunov theory.(2)For unknown continuous-time nonlinear systems with constrained input,an online policyiteration-based algorithm is employed to design the optimal event-triggered control law.Firstly,a novel identifier is proposed to make the estimation error converge quickly and the experience replay technique is employed to release the persistence of excitation condition.Then,the eventtriggered-based critic and actor networks are presented to approximate the cost function and the event-triggered control law,and the event-triggered-based integral reinforcement learning method is explored to solve the HJB equation.By utilizing the integral reinforcement learning,network resources are saved and the learning efficiency is improved.Based on the Lyapunov method,stability of the closed-loop system and the convergence property of the three networks estimation errors are proved.(3)For unknown discrete-time nonlinear systems,a problem of designing the optimal triggering policy and optimal event-triggered control law to optimize the performance index function consisted in the transmission cost is proposed and solved.At first,a model network is employed to approximate the system dynamic.Then,according to wether trigger or not,a switched system contained two subsystems is formulated,and the corresponding cost function and control law for the switched system are given.For the formulated switched system,the original trigger signal turns into the switching signal and the original optimal event-triggered control problem turns into the co-design problem of optimal control law and optimal switching law for the constructed switched system.In view of ADP-based optimal control methods for switched systems,critic and actor networks are applied respectively to approximate the optimal cost function and the optimal output feedback control law,and a policy-iteration-based online learning method is proposed to update weights of the networks.According to the Lyapunov stability,the stability of the closed-loop system and the convergent property of the proposed method are given.(4)For discrete-time switched nonlinear systems with bounded control input,an optimal event-triggered control problem is proposed at the first time.To solve this problem,an eventtriggered condition is designed to guarantee the stability of the closed-loop switched system.Then,event-triggered-based critic and actor networks are proposed and their weights are updated only at triggered instants,which not only decrease the the amount of computation but also reduce the transmission load.By the value iteration strategy,the optimal event-triggered control law and the optimal switching law can be obtained.However,it is noted that the computation amount is increased exponentially along with the iteration index and it takes a long time to obtain the optimal results.In order to take a shorter time to obtain the optimal results,an event-triggered-based integer-mixed ADP is proposed.Critic and actor networks are respectively used to approximate the event-triggered control law and the costate vector function for each subsystem.Thus,the number of critic and actor networks are the number of subsystems.At last,the policy iteration scheme is employed to derive the optimal switching law and the optimal event-triggered control law.In summary,for multiple classes nonlinear systems,optimal control laws are designed to optimize the performance index function based on ADP method,and the stability of the closedloop system and the convergence property of the proposed method are analyzed.