| In recent years,in order to reduce the consumption of fossil fuels and protect the ecological environment,China has vigorously promoted the development of electric vehicles,and the number of electric vehicles has grown rapidly.However,the unregulated access of a large number of electric vehicles to the power grid will cause new load peaks,resulting in problems such as decrease in the quality of electricity,increasing operating costs,and transformer overload.In order to alleviate these problems,coordinated charging algorithms have been applied to control the charging behavior of electric vehicles.However,existing rule-based coordinated charging methods can only be used in specific environments,and the algorithm’s effect is unstable,making it more suitable for static environments.On the other hand,coordinated charging methods based on objective optimization methods assume that all random information can be foreknown,which is unreachable in actual scenarios.In contrast to these two algorithms,model-free reinforcement learning algorithms can gradually learn optimal control decisions by repeatedly interacting with the environment without prior knowledge,and are more suitable for achieving the coordinated charging of electric vehicles that are full of randomness.Therefore,this thesis studies real-time control methods of coordinated charging for electric vehicles based on model-free reinforcement learning when the charging behavior is completely unknown.The main work of this thesis is as follows:(1)This thesis defines the Markov decision process of coordinated charging.The definition of the elements of the Markov decision process of coordinated charging is clarified,including states,actions,and rewards.And based on the real charging data of the Yuandayuan community in Beijing,the distribution functions of the arrival time,charging demand,and non-charging load of electric vehicles are fitted using a mixed Gaussian distribution.In addition,the impact of uncoordinated charging on the community power grid is studied using Monte Carlo sampling,and the results show that the power peak value increases from 336.67 kW to 435.28 kW during uncoordinated charging,which is a 29.29%increase,exceeding the safety limit of the community transformer and may cause damage to the transformer.(2)For the continuous control of charging power,an adaptive action exploration noise deep deterministic policy gradient algorithm is proposed.This thesis considers that fixed variance of exploration noise may cause the model to abandon better strategies and explore invalid strategies,and designs an adaptive exploration noise module,which allows the model to have different action exploration spaces at different training stages.In addition,considering the sparse reward problem faced in electric vehicle charging problems,this thesis further adds a priority experience replay buffer to the structure of the proposed algorithm,improving the learning ability and training speed of the algorithm.Finally,experiments were conducted in a simulation environment constructed based on a real environment.The experimental results show that compared with the deep deterministic policy control algorithm,the proposed algorithm can further reduce the standard deviation of the community total load by 0.74%and achieve better effect.(3)For the discrete control of charging power,this thesis proposes a two-stage dueling deep Q network algorithm combined with long shortterm memory networks.Firstly,considering the problem of the large dimension of the action space in the discrete control scenario,this thesis redesigns the action space of the electric vehicle charging problem and proposes a two-stage control method.Secondly,in order to address the problem of insufficient information representation,this thesis combines the deep Q network with the long short-term memory network to improve the stability and convergence of the algorithm.Finally,based on real charging data,simulation experiments were conducted,and the results show that the proposed algorithm can further reduce the load standard deviation by 3.43%,1.21%,and 1.33%,respectively,compared to the heuristic rule algorithm,the objective optimization algorithm,and the dueling deep Qnetwork algorithm. |
