Optimization and reinforcement learning techniques in multi-agent graphical games and economic dispatch

This work discusses optimization and reinforcement learning techniques in power system Economic Dispatch and Multi-Agent graphical games.;Power System Economic Dispatch (ED) is one of the power system energy management tools that is used to allocate required power generation to a number of generating units to meet the active load demand [109]. The operation cost of the power utilities depends on the fuel cost of the generating units. By optimizing the objective functions that depend on the fuel cost, the Economic Dispatch results in fuel cost savings, [25]. The generation cost functions are either smooth or non-smooth based on the nature of the generating units. One source of non-convexity is the physical constraints of the generation units such as spinning reserve, transmission losses, prohibited operation zones, ramp rate limit, valve point loading effect, and multiple fuel options [109]. Besides, some generating units have multiple steam valves, which open in a sequential manner. This introduces mathematical difficulty to the generation cost function by adding the effect of the ripples to the generation cost function [4], [5], [56]. This makes the Economic Dispatch problem a large-scale nonlinear constraint optimization problem.;The dynamic graphical game results from multi-agent dynamical systems, where it is desired to make all the agents synchronize to the state of a command generator or leader agent, the interactions between agents are prescribed by a communication graph structure. Cooperative control refers to a dynamical systems interconnected by a communication graph. Synchronization allows each agent of the cooperative team to reach the same state by the proper design of decision and control protocols. In multi-player cooperative games Nash solutions relies on solving coupled Hamilton Jacobi equations. The result is the Nash equilibrium solution.;Contributions to Economic Dispatch The contributions of this research to the power system Economic Dispatch can be summarized as follows.;A dynamic formulation technique is developed to efficiently allocate the change in the total load demand to the generating units. This technique is shown to be insensitive to the initial load distribution and it guarantees an optimal distribution among the generating units. A new approximation of the non-convex cost function is developed to solve non-convex Economic Dispatch problem with the consideration of transmission losses. This approximation enables the use of gradient and Newton Raphson techniques to solve the non-convex Economic Dispatch problem with valve point loading effect and transmission losses. Moreover, ideas from Reinforcement Learning (RL) are used to solve the non-convex Economic Dispatch problem [82], [38].;The contributions of this research to the multi-agent graphical games can be summarized as follows Contributions to multi-agent graphical games A new class of multi-agent discrete-time games known as dynamic graphical games is developed. The agents' error dynamics are coupled dynamical systems driven by the control of each agent and its neighbors. A performance index is defined for each agent that also depends only on the local neighbors of each agent. A new notation of interactive Nash equilibrium is introduced which holds if all agents are in Nash equilibrium and the graph is strongly connected. Graphical games Bellman equations are derived and shown to be equivalent to certain graphical game Hamilton Jacobi Bellman equations developed herein. Nash solutions and best response solutions are given in terms of solutions to the discrete time Hamilton Jacobi Bellman equations. Reinforcement Learning (RL) techniques are used to solve these dynamic graphical games. A set of coupled Riccati recursions will be derived to provide offline solutions for the dynamic graphical game. Offline and real-time actor-critic network schemes are developed to implement the proposed value iteration algorithms.;Approximate Dynamic Programming (ADHDP) or Q learning is used to solve the dynamic graphical game, where the dynamics of the agents are not required. In the Q-learning approach, a parametric structure is used to approximate the Q-function of the control policy of each agent. The Coupled difference Riccati equations associated with the linear quadratic discrete-time dynamic graphical games are given.;Furthermore, the notion of differential graphical games is developed for continuous-time multi-agent systems, where the performance index for each node depends only on local neighbor information. Synchronization in differential graphs is studied using continuous-time error dynamics for each node. The error dynamics for each node are influenced by its own actions and those of its neighbors. Graphical game Integral Reinforcement Learning (IRL) Bellman equations are derived and shown to be equivalent to certain graphical game IRL Hamilton Jacobi Bellman equations developed herein. Nash solutions and best response solutions are given in terms of solutions to continuous-time IRL HJB equations. Policy iteration convergence results are given throughout. Finally, integral reinforcement learning structures are developed to solve the dynamic graphical game using policy iteration.