Controller Synthesis For Intelligent Systems Based On Meta-Reinforcement Learning

Recently,intelligent control systems have been widely used in autonomous vehicles,unmanned aircraft,intelligent robots,intelligent medical systems and other safetycritical systems.Any mistake may cause serious impact or even disastrous consequences.Therefore,it has become an urgent problem to verify whether the intelligent control system can meet its requirements.In recent years,with the rapid development of deep learning,there are more and more researches on using deep neural network(DNN)to control safety-critical systems.For these safety-critical systems,one of the most important and challenging problems is safe controller generation,that is,to construct a neural network controller that can ensure that the trajectory of the system does not intersect with the unsafe region.In order to ensure the safety performance of synthetic DNN controller,a lot of work has been focused on the safety verification of DNN control closed-loop system,which is a very difficult problem because DNN expressions are often highly nonlinear.Aiming at the problems and challenges of intelligent system controller generation,this paper studies the construction method of deep neural network(DNN)controller in continuous system with security constraints.From the aspects of scalability and high efficiency,three different DNN controller generation methods are proposed.The following contributions have been made:(1)Firstly,this paper proposes a controller construction method with reinforcement learning DDPG algorithm as the core.The method uses the iterative generation method to make the learning component Learner and the verification component Verifier interact to synthesize the safe DNN controller.Among them,Verifier trains DNN controller through deep reinforcement learning,and Verifier proves the learned controller by calculating the maximum safe initial region and its corresponding obstacle function based on polynomial abstraction and bilinear matrix inequality solution.Compared with the existing controller generation and verification methods,it can adapt the barrier function without the need to learn through artificially defined functions to secure the controller.Experimental results show that the method based on numerical optimization is more effective than the one based on satisficability mode theory(SMT),and can efficiently synthesize a safe DNN controller and deal with nonlinear systems with dimensions up to 12.(2)Further,this paper proposes a meta-reinforcement Learning DNN controller construction method by combining DDPG controller synthesis method with meta-learning algorithm Model-Agnostic meta-learning(MAML).This method can customize the meta-network of this dimension for the continuous system controller.By using the meta-network as the initial parameter of DDPG algorithm training,all the continuous system controller training tasks of the same dimension can obtain DNN controller in less time and shorter iterations.This greatly improves the efficiency of DNN controller acquisition.Experimental results show that this method can synthesize safe DNN controller effectively,and is more efficient than the basic DDPG synthesis method.(3)Based on the above two methods,this paper further generalizes the method and proposes a unified meta-reinforcement learning DNN controller construction method.This method unifies low-dimensional and high-dimensional continuous system controllers so as to solve the limitation that same-dimensional DDPG network structure can only train same-dimensional controllers.It breaks the shackles of retraining complete DNN controller when facing different intelligent systems,and greatly improves the generalization of deep neural network(DNN)controller construction method.Experimental results show that this method can synthesize effective and unified DNN controller meta-network,which can be used as the initial parameters of DNN controller for different dimensions and different systems,making the system controller generation more efficient and unified in terms of performance and generation method.