Research On Application Of Deep Reinforcement Learning In Strategy Optimization Of Electricity Market Participants

In recent years,with the continuous development of China's new round of power system reform,the structure and operating mechanism in the electricity market have been obviously changed.As a market subject in the new environment,it has become an inevitable trend for power retailers to participate in market deal in various ways to maximize profits.Therefore,a reasonable strategy for electricity procurement and marketing is the core issue in the decision of the retailers.At the same time,the widespread application of distributed energy resources provides a channel for users to participate in market with excess electricity.In this context,studying the behavior laws of electricity users has important meanings to the improvement of the market.However,with the deepening of reform,the uncertainty and volatility of the electricity market have increased significantly,which poses a huge challenge for optimizing the strategies of market participants.Thus,in the face of multiple uncertainties caused by random energy,loads,and sundry prices,how to establish more general operating models for various participants based on current market rules,choose scientific decision-making methods to optimize their behavior strategies,and maximize the benefits has become a significant problem in the electricity market field.At present,researchers have proposed a series of effective optimization methods.Among them,scenario-based stochastic optimization method usually generates random scenarios by presetting the probability distribution of uncertain variables,which can cope with the complex market environment.However,this approach has problems of long calculation time,large probability distribution errors,and high model dimensions.Robust optimization method only considers the extreme results of the uncertainty,and has advantages of high computational efficiency and less required information.But this approach ignores the probability distribution information of uncertain variables,often leading to conservative decisions.Interval optimization method describes the uncertainty through precise probability distribution model,and chooses the confidence interval to ensure reliability of the decisions.However,there are difficulties in the probability modeling of the uncertainty and the selection of confidence interval.In addition,machine learning algorithm has been widely applied to the electricity market.In this context,this thesis does research on the behavior strategies of electricity market participants based on deep reinforcement learning.Deep reinforcement learning integrates the ideas of deep learning and reinforcement learning,and shows advantages in dealing with high-dimensional uncertain problems.This data-driven model-free algorithm can fully mine the effective information in the sample,avoiding the artificial assumption of the uncertainty,and further improving the effectiveness of decisions.The main content and results of this thesis are as follows:This thesis first describes the optimization of electricity market trading behavior as a Markov decision process,and introduces the learning mechanism of market participants in the framework of deep reinforcement learning.Next,in view of the issues that low computational efficiency,over-conservative decisions,and large modeling errors of traditional optimization methods,a load declaration method for power retailers in the spot market based on deep Q network is proposed.The approach is directly driven by big data and does not require the assumption of the uncertainty in the optimization process,which avoids modeling error.Meanwhile,this algorithm utilizes deep neural networks to approximate the optimal behavior strategy under uncertainty and improves computational efficiency.Moreover,based on deep Q network,this thesis integrates the algorithms such as dueling network,double Q learning and prioritized replay buffer.Finally,the results verify the effectiveness of the improved algorithm in coping with uncertainty.Furthermore,considering the shortcomings of traditional methods and the errors caused by discretization operation of deep Q network,this thesis presents an optimization model for the battery swapping station based on deep deterministic policy gradient.Among them,in order to reduce the economic losses due to uncertain prices and battery demand,this policy-based algorithm is first introduced to continuously control the real-time charging/discharging power of multiple charging piles.Besides,this thesis also explores the algorithmic mechanism and parameter setting of deep deterministic policy gradient to further improve optimization effect.By comparing with other methods,the superiority of the proposed model in optimizing the behavior of electricity market participants is verified.