Transfer Reinforcement Learning For Power System Optimization

Nonlinear programming is ubiquitous in power system operation,such as units commitment,reactive power optimization,etc.To handle these issues,the common methods include conventional mathematical approaches and artificial intelligence(AI)techniques.However,the conventional mathematical approaches,e.g.,Newton method,quadratic programming,interior-point method,are highly dependent on an accurate mathematical model and may merely obtain a local optimum if system nonlinearities,discontinuous functions and constraints,and functions with multiple local-minima exist.In contrast,AI algorithms,such as artificial bee colony,ant colony system,particle swarm optimization,genetic algorithm,are highly independent on the mathematical model,which have been applied for the optimal operation of power systems.Unfortunately,most of these approaches is incapable of recording the prior knowledge,which results in a relatively long computation time when dealing with a new optimization task.Consequently,it is difficult to achieve a fast dynamic optimization of a large-scale power system which optimization tasks may vary along with the time.Hence,this paper proposes a novel transfer reinforcement learning(TRL)for a fast optimization of large-scale power systems.The swarm intelligence is adopted for an efficient exploration and exploitation in an environment,which can simultaneously update more than one element of the common knowledge matrices.The optimal knowledge matrices of previous source tasks can be stored and used for constructing an approximate optimal knowledge matrices of each new task,thus the convergence rate can be accelerated.For a large-scale optimization,the whole system will be decomposed into several small-scale subsystems,in which the game theory and consensus theory are introduced for achieving a coordination between different subsystems in the upper-layer optimization,while TRL or consensus theory are employed for rapidly obtaining an optimal solution in the bottom-layer optimization.In summary,the optimization theory system of TRL will be gradually presented for different optimizations of power systems,as follows:1)A novel centralized TRL with a single continuous source task is proposed.The associative memory is presented to handle the curse of dimension resulted from the knowledge matrix with multiple controllable variables.To accelerate the knowledge matrices update,the swarm intelligence based on the cooperative mechanisms of ant colony and bee colony is employed for a proper balance between exploration and exploitation.Besides,the optimal knowledge matrices of the last optimization task will be directly transferred to the initial knowledge matrices of the current new task,thus a blind search without any prior knowledge can be effectively avoided,which can lead to a fast optimization speed.Finally,the optimization performance of two proposed algorithms are thoroughly tested on the classical reactive power optimization of power systems.2)A novel linear TRL with multiple source task is proposed.The binary associative memory is introduced to address the optimization with continuous controllable variables,while the imitation learning is used for accelerating the exploration and exploitation in the initial phase of constructing the knowledge matrices.The interactive coordination between different agents is achieved by the consensus theory.The approximate optimal knowledge matrices of a new task can be linearly generated with a high precision by exploiting the optimal knowledge matrices of the source tasks according to their similarities.Finally,the optimization performance of the proposed single-agent TRL and multi-agent TRL are thoroughly tested on centralized and decentralized dynamic generation command dispatch of automatic generation control(AGC),respectively.3)A novel nonlinear TRL with multiple source task is proposed.The coordinations between different agents are realized by Nash equilibrium and Stackelberg equilibrium game theory,respectively.The scale-limited source tasks base is replaced by extreme learning machine and deep belief network,respectively,thus the storage of optimal knowledge matrices can be reduced significantly,while the initial knowledge matrices of the new task can be more closer to its optimal knowledge matrices.Finally,the proposed two multi-agent TRL techniques are investigated on decentralized optimal carbon-energy combined-flow and real-time optimal dispatch between supply-side and demand-side of power systems,respectively.