Research On Reinforcement Learning Based On Active Disturbance Rejection Control For Nonlinear System

The control of unknown nonlinear systems has always been an important issue in the field of system control.Since the system dynamics are unknown,many control algorithms based on mechanism models cannot be implemented.Reinforcement learning can learn from interactions with unknown environment,and prior knowledge of system structure information is unnecessary.Therefore the control method based on reinforcement learning becomes an ideal algorithm for unknown nonlinear system control.In recent years,reinforcement learning has achieved many results in the field of nonlinear system control.However,most nonlinear systems in reality are often affected by various types of disturbances.The disturbance rejection ability of reinforcement learning is not powerful enough.The system output often deviates greatly from the set value under disturbance,and the system performance degrade severely.This paper studies how to improve the control precision and disturbance rejection ability of reinforcement learning in nonlinear system control.Since most state variables of actual nonlinear systems are continuous,the continuous reinforcement learning algorithm in the actor-critic network structure is used in nonlinear system control most widely.There are some problems with this algorithm in application:1.When the reference trajectory changes continuously,especially when the trajectory has a large curvature,the tracking error becomes large.2.The disturbance rejection ability of the algorithm is very limited.As the disturbance gradually increases,the control effectiveness sharply declines until it diverges.To solve these problems in unknown nonlinear systems control,this paper first establishes a reinforcement learning framework in actor-critic network structure.To avoid using system structure information,a certain performance index function is used.1.In order to improve the ability to track reference value trajectories of control system,especially when the trajectories continuously change drastically,a factor which can dynamically adjust the weight update law of the actor network is proposed.So that the controller can adjust the control strategy more timely when the reference trajectory frequently changes.2.In order to improve the disturbance rejection ability of the control system,an extended state observer is used to convert the unknown external disturbance and internal parameter perturbation to an overall disturbance,so that the control system can reduce the impact of disturbances on system.3.Furthermore,we combined the reinforcement learning algorithm with the linear active disturbance rejection,replace the original actor neural network with an active disturbance rejection controller.This method can greatly improve the system's disturbance rejection ability and control precision.At the same time,the number of parameters to be tuned no longer increases exponentially with the input dimension.In order to verify the effectiveness of the improved reinforcement learning algorithm,the tracking control of an unknown nonlinear pure feedback system are used as a benchmark.The principles and design of reinforcement learning-extended state observer(RL-ESO,based on improvement 1 and 2)algorithm and reinforcement learning-active disturbance rejection(RL-ADRC,based on improvement 3)algorithm are also given.In the pure feedback system tracking control experiment,the reference trajectory is a smooth trajectory with periodic changes,and the external disturbances are continuously changing.The controller only obtains the system output and control inputs without any other structure information.Analyze and compare the performance between the two improved algorithms and the original algorithm.The results show that compared with the original algorithm,RL-ESO has a higher control precision and a stronger disturbance rejection ability,RL-ADRC not only has a stronger disturbance rejection ability,but also has a faster training speed.