| The field of adaptive dynamic programming and its applications to control engineering problems has undergone rapid progress over the past few years. Recently, a new theory called Robust Adaptive Dynamic Programming (for short, RADP) has been developed for the design of robust optimal controllers for linear and nonlinear systems subject to both parametric and dynamic uncertainties. This dissertation integrates our recent contributions to the development of the theory of RADP and illustrates its potential applications in both engineering and biological systems.;In order to develop the RADP framework, our attention is first focused on the development of an ADP-based online learning method for continuous-time (CT) linear systems with completely unknown system. This problem is challenging due to the different structures between CT and discrete-time (DT) algebraic Riccati equations (AREs), and therefore methods developed for DT ADP cannot be directly applied in the CT setting. This obstacle is overcome in our work by taking advantages of exploration noise. The methodology is immediately extended to deal with CT affine nonlinear systems, via neural-networks-based approximation of the Hamilton-Jacobi-Bellman (HJB) equation, of which the solution is extremely difficult to be obtained analytically. To achieve global stabilization, for the first time we propose an idea of global ADP (or GADP), in which we relax the problem of solving the Hamilton-Jacobi-Bellman (HJB) equation to an optimization problem, of which a suboptimal solution is obtained via a sum-of-squares-program-based policy iteration method. The resultant control policy is globally stabilizing, instead of semi-globally or locally stabilizing.;Then, we develop RADP aimed at computing globally stabilizing and suboptimal control policies in the presence of dynamic uncertainties. A key strategy is to integrate ADP theory with techniques in modern nonlinear control with a unique objective of filling a gap in the past literature of ADP without taking into account dynamic uncertainties. The development of this framework contains two major steps. First, we study an RADP method for partially linear systems (i.e., linear systems with nonlinear dynamic uncertainties) and weakly nonlinear large-scale systems. Global stabilization of the systems can be achieved by selecting performance indices with appropriate weights for the nominal system. Second, we extend the RADP framework for affine nonlinear systems with nonlinear dynamic uncertainties. To achieve robust stabilization, we resort to tools from nonlinear control theory, such as gain assignment and the ISS nonlinear small-gain theorem.;From the perspective of RADP, we derive a novel computational mechanism for sensorimotor control. Sharing some essential features of reinforcement learning, which was originally observed from mammals, the RADP model for sensorimotor control suggests that, instead of identifying the system dynamics of both the motor system and the environment, the central nervous system (CNS) computes iteratively a robust optimal control policy using the real-time sensory data. By comparing our numerical results with experimentally observed data, we show that the proposed model can reproduce movement trajectories which are consistent with experimental observations. In addition, the RADP theory provides a unified framework that connects optimality and robustness properties in the sensorimotor system. Therefore, we argue that the CNS may use RADP-like learning strategies to coordinate movements and to achieve successful adaptation in the presence of static and/or dynamic uncertainties. |
