Research On Reinforcement Learning-Based Adaptive Tracking Control

Recent years have witnessed the adaptive control development since artificial intelligence has been developed and flourished.In particular,reinforcement learning has been studied extensively,which can introduce the long-term performance index into the control design process.It is well known that there exists a variety of practical operating conditions,such as existing unmodeled dynamics and external disturbances,input saturation,prescribed transient performance,unknown control directions,limited communication transmission bandwidths,limited computational capabilities,limited energy supplies,etc.Besides,it is very difficult or even impossible to obtain accurate system models.What’s more,the operation region has become larger while the requirement of control performance has been improved.Due to the universal approximated property,the neural network has been widely used to deal with the unknown systems dynamics.Under different operation conditions,it is complicated to design the adaptive control,which not only can ensure the closed-loop stability and convergence,but also can meet the long-term control performance.It is worth studying the combination of adaptive control,neural network and reinforcement learning,since it has profound theoretical significance and practical engineering value.Therefore,this dissertation focuses on studying the reinforcement learning-based adaptive tracking control under practical operation conditions,while the main contributions can be provided as follows:(1)For continuous-time nonlinear systems with external disturbances,reinforcement learning-based optimal control is presented to address the two-player zero-sum game with partially unknown system dynamics and input constraints.The control optimality is for the case that the virtual disturbance is at the worst case and then the proposed control can be used to deal with practical disturbances.Three neural networks are given to approximate value function,control policy and disturbance policy,respectively.The synchronous update algorithm is proposed for the three neural networks and a new function is introduced into the update laws of actor and disturbance neural networks.It can ensure that the proposed control can work well even if the initial control policy is unstabilized.The proposed control will converge to its optimal value if the persistent excitation condition is satisfied.Applying Lyapunov stability theory,the state,additional function gradient and the weight errors are all uniformly ultimately bounded.Simulation results have verified the effectiveness and optimality of this control method.(2)For multi-input multi-output continuous-time nonlinear systems,nonlinear sliding mode-based adaptive tracking control is proposed with partially unknown system dynamics and unmeasured velocities.This nonlinear sliding mode not only can amplify errors if they are very small,but also can saturate errors in a reasonable region if they are larger than the prescribed thresholds.Due to the universal approximation property,a neural network is established to approximate the unknown system dynamics,while a first-order robust exact differentiator is designed to estimate the unmeasured velocities.Applying this control method,the closed-loop stability can be ensured and the tracking errors will converge to zeros.Then,simulation experiments are performed based on ODIN system model and then simulation results demonstrate the effectiveness of this control method.(3)For continuous-time nonlinear systems with prescribed transient performance,reinforcement learning-based adaptive tracking control is studied with partially unknown dynamics and input constraints.Through the error transformation,the state-constrained control issue can be transferred into an unconstrained one.Two neural networks are constructed to approximate the long-term performance index and the unknown dynamics,respectively.An auxiliary compensator is designed to deal with the input saturation issue.With this control method,it not only can ensure that all closed-loop error signals are all uniformly ultimately bounded,but also can satisfy the prescribed transient performance and input saturation,while the long-term performance index is minimized simultaneously.Then,simulation results can demonstrate good tracking performance and the prescribed transient performance can be satisfied.(4)For continuous-time nonlinear systems with unknown control directions,reinforcement learning-based adaptive tracking control is proposed with completely unknown system dynamics.Nussbaum function is applied to deal with unknown control directions and two neural networks are established to approximate the long-term performance index and unknown dynamics,respectively.The synchronous update algorithm is designed based on σ-modification.Then,it can be ensured that the closed-loop signals are all uniformly ultimately bounded,while the long-term performance index can be minimized at the same time.The effectiveness of this control method is demonstrated by simulation results.(5)There are a variety of practical operation conditions,such as limited communication transmission bandwidths,limited computational capabilities and limited energy supplies.Considering these operation conditions,event-triggered reinforcement learning-based adaptive tracking control is studied for completely unknown dynamics systems.Two neural networks are used to approximate the long-term performance index and controller,respectively.The event-triggered condition is provided and weights of the two neural networks can be held by applying Zero-Order Holder technique during the event-triggered interval.The weights will just be updated at the event-triggered instant.It can be ensured that tracking errors and weight errors are all uniformly ultimately bounded,which the Zeno phenomenon can be obviated simultaneously.Then,the proposed control performance has been verified by simulation results.