Reinforcement Learning Based Intelligence Frequency Regulation For V2G Integrated Power Systems

The rapid development of the global economy has consumed a large amount of nonrenewable energy,which has triggered global problems such as climate warming,resource shortage and atmospheric pollution.To alleviate these problems,countries are actively developing renewable energy sources.However,the output power of renewable energy is highly influenced by the environment,resulting in intermittent output power of renewable energy.This triggers an imbalance between the generated power and the electric load,and also makes the grid frequency regulation difficult.Therefore,energy storage systems are used as an auxiliary frequency regulation solution to solve the power supply-demand imbalance,but energy storage systems require significant construction costs.The decline in oil reserves and the government’s accelerated promotion of energy conservation and emission reduction have led to a rapid growth in the ownership of electric vehicles.This has given rise to the development and application of Vehicle to Grid(V2G)technology,which provides a reliable way to store energy and an economical and environmentally friendly frequency regulation scheme.However,integrating V2 G systems into the grid also brings many new features and challenges that traditional controllers may not be able to cope with.Therefore,in this thesis,the characteristics and roles of V2 G participation in grid frequency regulation are verified and analyzed with the theme of improving grid frequency regulation performance using V2 G systems.The characteristics and challenges brought by V2 G participation in frequency regulation are studied and analyzed,and to solve these new problems,a cooperative integral reinforcement learning algorithm and attack intensity aware algorithm are proposed to improve the frequency regulation performance of power systems.The main work of this thesis is as follows:1.In this thesis,a grid frequency regulation model and a state space model of the electric vehicle considering the expected demand of the electric vehicle are developed for the V2 G integrated power system.The Proportional-Integral-Derivative(PID)controller is applied to the grid frequency regulation to verify the correctness of the grid frequency regulation model.The advantages of V2 G participation in frequency regulation are demonstrated through simulation experiments,and it is also demonstrated that the State of Charge(SOC)of the battery can be maintained in a good state during the provision of frequency regulation service by the EV using the SOC retention algorithm.2.To address the problem of lack of collaboration between power plant turbine and V2 G system in microgrid frequency regulation service,an intelligent frequency control algorithm is proposed in this thesis.The algorithm is based on cooperative integral reinforcement learning for controlling and coordinating the frequency regulation services of turbine and V2 G systems.The parameters of each controller are adjusted by cooperative integral reinforcement learning to facilitate the collaboration between the turbine and V2 G systems,thus minimizing the frequency deviation.In addition,the algorithm takes into account the characteristics of different controllers,and the data-driven nature of the algorithm plays to the strengths of each controller,ultimately minimizing the total control cost.A novel switching neural network is also proposed to address the ”error learning”problem of Reinforcement Learning(RL)in V2 G systems with asymmetric capacity constraints.The effectiveness of the switching neural network in solving the ”error learning”problem and the superiority of the proposed algorithm are demonstrated in the microgrid simulation experiments.3.Distributed electric vehicles require extensive communication networks to connect to V2 G controllers,but network attacks may pose a threat to network security and thus degrade the performance and stability of the frequency regulation system.To address this problem this thesis proposes a novel V2 G frequency regulation algorithm,which is based on an attack intensity-aware adaptive critic design,for improving the frequency regulation performance of V2 G integrated grids when subjected to Denial of Service(DoS)attacks.The algorithm uses reinforcement learning methods to solve the optimal robust V2 G frequency regulation controller in a game-theoretic framework to mitigate the impact from load,renewable energy and DoS attacks.To cope with denial of service attacks,we construct an attack intensity-aware reinforcement learning model in which the cost function is able to establish a mapping between system state,attack intensity,and frequency tuning performance.This enables the controller to automatically adjust the control output according to the current attack intensity to cope with denial-of-service attacks.We use reinforcement learning to optimize the performance of the controller so that it can adaptively adjust the control strategy in response to different attack intensities and environments to ensure the stability and reliability of grid operation.The frequency regulation controller trained by reinforcement learning can effectively mitigate the impact from load,renewable resource,and denial-of-service attacks,and improve the frequency regulation performance and robustness of the electric vehicle integrated grid.Simulations on the IEEE 39 bus power system containing three areas validate the effectiveness and advantages of the proposed algorithm under denial-of-service attacks.