Deep Reinforcement Learning Based Auxiliary Computing Method For Power System Operation State

The Operation State Calculation(OSC)is an essential theoretical basis for dispatching a power grid safely and stably.Usually,the annual OSC should be conducted based on the next year's grid planning and load forecasting.The Typical Operation States(TOSs)are preliminarily formulated referring to historical experience,based on which the safe operation bounds of a power system can be further obtained by stability calculation.Now,in practice,the OSC is still coordinately completed by a great deal of workforce from sub-dispatching centers with power system simulation software(such as PSASP,PSD-BPA).With the launch of the new generation ultra-high voltage AC/DC simulation platform of the state grid in 2017,the OSC has stepped up to a new level,and the overall simulation capability and work efficiency have improved significantly.However,in recent years,the operation states of the state grid are becoming increasingly complex and changeable,as a result of the grid's fast expanding,the gradual formulation of the AC/DC hybrid pattern and the accession of a high proportion of renewable energy.The OSC still depends to some extent on expert experience.To further improve the automatic analysis capability of the auxiliary simulation software and OSC work efficiency,it is urgent to enhance the automation level of the OSC process.The Power Flow Convergence(PFC)adjustment and the Tie-line Power(TP)adjustment of a Key Transmission Section(KTS)are the most massive and most repetitive tasks in the OSC.Automatically completing these two parts means a lot to the automation of the whole OSC process.In this paper,the experts' adjusting process is first analyzed and simulated,based on which the mathematical models are formulated for the PFC and TP adjustment.Then,the Improved Deep Reinforcement Learning(IDRL)method is proposed for training.Finally,the automatic PFC and flexible TP adjustment are achieved with the trained Deep Neural Networks(DNNs).The research details in this paper are as follows:(1)A Markov Decision Process(MDP)is formulated for the PFC adjustment.The practical PFC adjustment is studied,and an improved mapping strategy considering the network loss is proposed.Then,the PFC adjustment is formulated as an MDP,which suits the computer simulation well.(2)An improved deep Q learning method is proposed for the formulated MDP in(1)to automate the PFC adjustment.In practice,the PFC adjustment usually refers to only turning on/off the generators,which are discrete variables.Thus,an improved value-based deep Q learning method is proposed to solve the MDP in(1)and verified on the IEEE-118 bus system and an actual power grid,respectively.(3)An MDP is formulated for the TP adjustment of a KTS.The TP adjustment in practical projects is first analyzed,and an improved mapping strategy combining experts' experience is proposed,with which the TP adjustment is formulated as an MDP that well suits the computer simulation.(4)An Improved Deep Reinforcement Learning(IDRL)method based on Actor-Critic structure is proposed for the MDP in(3)to achieve the flexible TP adjustment of a KTS.Since the TP of a KTS is a continuous variable,an IDRL approach that suits the continuous adjustment is proposed.(5)The proposed method in(4)is prone to failure when there are too many KTSs or the TP ranges are too large,for the reason that the fitting ability of a specific DNN is limited.To avoid redesigning more complex DNNs,the"stepwise training" and "prioritized target replay" are presented and verified on the IEEE-39 bus system and an actual power grid,respectively.Finally,the related parameters setting is discussed for the training efficiency improvement.