Research On Integrated Energy Multi-micro Grid System Optimization Strategy Based On Hierarchical Constraint Reinforcement Learning

The muti-microgrid system,comprising renewable energy,multi-energy loads,and distributed energy storage,has become a promising solution to increase renewable energy consumption and enhance the stability of the power grid.Nonetheless,the current optimization problems in muti-microgrid systems exhibit diversity and complexity,along with the uncertainties of power generation and load,as well as the privacy protection of microgrids,resulting in significant challenges for efficient model solving.Therefore,it is necessary to find an optimization strategy that can achieve efficient energy management to improve system performance.This paper proposes a hierarchical constraint reinforcement learning optimization approach for integrated energy microgrid systems to address these issues.First,a hierarchical constraint reinforcement learning optimization model for multi-microgrid is constructed.To solve the constraint violation problem that traditional reinforcement learning has difficulty in handling,the integrated energy multi-microgrid system optimization problem with constraint on states and actions is modeled by constrained Markov processes.The constraint optimization problem is transformed into an unconstrained optimization problem through Lagrangian relaxation.At the same time,the optimization problem of multiple microgrids is designed hierarchically to simplify the solution.The upper-level agent fully considers the time-relatedness and comprehensive cumulative return of the entire decision cycle,formulating the interaction strategy between microgrids and the energy storage optimization strategy based on the predicted net load and energy storage status information of each microgrid and issuing it to the lower level.The lower-level microgrids use the upper-level strategy as constraints and employ mathematical programming to solve the optimal output of internal devices while transmitting reward signals to guide the upper-level strategy update to avoid ineffective exploration of the upper-level intelligent agent.By coordinating the upper and lower levels,the proposed approach achieves global optimization of the microgrid system.Furthermore,a hierarchical constraint reinforcement learning solution method is introduced to solve the model.The method combines deep reinforcement learning and Lagrange multiplier methods and utilizes primal-dual optimization techniques for policy updates to guide the agent towards finding the optimal policy while adhering to the constraints.The proposed method fully utilizes the ability of reinforcement learning for adaptive decision-making,while effectively balancing the solution accuracy of mathematical programming,and also does not require aggregation of all microgrid state information,which effectively protects data privacy and solves the constraint problem that traditional reinforcement learning has difficulty in handling.Finally,numerical simulations are performed to compare the proposed approach with traditional centralized optimization and reinforcement learning methods,demonstrating its effectiveness and superiority in optimizing the microgrid system.