A Research On Adaptive Dynamic Programming Based Distributed Control And Its Application

In recent years,in view of the urgent demand for economic scheduling,resource al-location,network layout and other optimization problems of complex systems,distributed control has become a research hotspot in the field of control science.But the distributed optimal control problem needs to solve the Hamilton Jacobi Bellman(HJB)equation,which leads to the ‘curse of dimensionality' problem.Adaptive dynamic programming(ADP)integrates the theories and methods of dynamic programming,reinforcement learn-ing,neural network,adaptive control,optimal control and so on.It is one of the effective methods to solve the problem of ‘curse of dimensionality'.Therefore,this paper uses ADP method to study the tracking control,containment control,antisynchronization control of multi-agent system and its application in the assistant walking control of lower limb ex-oskeleton robot.Aiming at these control problems,we propose a distributed controller design method with unknown system model,and proposed new ADP algorithm and net-work implementation structure design to improve the control performance of the system.The main research results of this dissertation are as follows:1.An optimal tracking control problem is studied for discrete-time multi-agent sys-tems.A novel ADP algorithm is developed,compared to the classical policy iteration ADP algorithm,a two-stage policy iteration algorithm is proposed to obtain the iterative control laws and the iterative performance index functions.The proposed algorithm con-tains a sub-iteration procedure to calculate the iterative performance index functions at the policy evaluation.The convergence proof for the iterative performance index func-tions and the iterative control laws are provided.The stability of the closed-loop error system is provided.Further,an actor-critic neural network is used to approximate both the iterative control laws and the iterative performance index functions.The actor-critic neural network can implement the developed algorithm online without knowledge of the system dynamics.Compared with the traditional multi-agent distributed tracking control method,the proposed method can realize online learning and control of the system under the condition of unknown system dynamics,and meet certain optimization indexes.2.An optimal containment control of discrete-time multi-agent systems and continu-ous time multi-agent systems with system disturbances are studied.For the first problem,we integrates graph theory,optimal control theory and ADP to solve the optimal contain-ment control.Firstly,we introduce the containment error and corresponding discounted performance index function,and transform the traditional containment control problem to optimal control problem by using Bellman optimality principle.Secondly,the cou-pled discrete-time HJB equation has been derived for the optimal containment control,and the multiple agent based value iteration heuristic dynamic programming algorithm is introduced to solve the coupled discrete-time HJB equation indirectly.Moreover,the con-vergent proof and stability analysis of the proposed algorithm are provided.The optimal containment control problem with system disturbance,the robust control problem of the original system is transformed into the optimal control problem of the auxiliary system,and a distributed auxiliary optimal controller design is proposed.Then,the equivalent relationship between the original control problem and the auxiliary control problem is established.Finally,the neural network approximation framework is used to realize the online solution of the control algorithm.3.An output antisynchronization problem of multi-agent systems is considered by using an input-output data-based reinforcement learning approach.Most of the existing results on antisynchronization problems required full state information and exact system dynamics in the controller design,which is always invalid in practical scenarios.To ad-dress this issue,a new system representation is constructed by using just the available input/output data from the multi-agent system.Then,a novel value iteration algorithm is proposed to compute the optimal control laws for the agents? moreover,a convergence analysis is presented for the proposed algorithm.In the implementation of data-based controller,an incremental neural network structure is proposed to learn the optimal con-trol law.The proposed system modeling and control design only rely on the measurable system input/output data.The proposed network parameter update rules can improve the learning efficiency of the controller and enhance the application value of distributed con-trol in practical engineering.4.An ADP-based human-machine cooperative control problem is studied for walk-ing assistance lower limb exoskeleton(LLE).LLE has gained considerable interests in walking assistance applications for both paraplegia and hemiplegia patients.In walking assistance of hemiplegia patients,the exoskeleton should have the ability to control the affected leg to follow the unaffected leg's motion naturally.In order to ensure that the controller adapts to different wearers,we present a ADP-based human-machine cooper-ative control strategy.In the proposed control strategy,we modeled the control system of lower exoskeleton as a leader-follower multi-agent system.Then,an online learning reinforcement learning framework is proposed,a policy iteration algorithm is employed to achieve better tracking control performance for lower exoskeleton.Finally,the neural network implementation of cooperative control algorithm for lower limb exoskeleton is given.The proposed method overcomes the dependence of the traditional walking assis-tance control method on accurate system dynamics modeling,and improves the adaptabil-ity of exoskeleton robot to wearer.