Traffic Signal Timing For Regional Boundary Intersections Using Deep Reinforcement Learning

With the rapid development of urbanization,the problem of urban traffic congestion has expanded from nodes and arteries to regions.As a result,regional traffic control is significantly necessary for alleviating urban traffic congestion.The macroscopic fundamental diagram（MFD）is a graph reflecting the relationship among regional vehicle flows,densities and speeds,which can be utilized for evaluating the network performance.Using MFD to manipulate the traffic flow at the perimeter of a congested region（i.e.,perimeter control）,so as to alleviate the congestion in such a protected region,is a research hotspot in the field of macroscopic traffic control.From a microscopic perspective,signal control at intersections is an essential method to implement regional perimeter control.Most existing signal control methods adjust signal timings based on historical data,which has limitations of responsing to dynamic traffic states.However,signal control based on deep reinforcement learning has the characteristics of selflearning and adapting to random traffic environment,which is more advantageous than traditional signal control.As a result,this thesis adopts MFD for traffic flow control among regions,while deep reinforcement learning algorithms are applied to signal timing optimization at boundary intersections.The effectiveness of the proposed macro-micro perimeter control strategy is verified by a traffic simulation.The research contents of this thesis are given as follows:First,the macroscopic perimeter control parameters are solved based on a model predictive control framework.For a two-region traffic system,a regional perimeter control model is established,in which the boundary vehicle release ratio is the control parameter and the maximization of regional completion flow is the objective.The regional perimeter control model is used to predict the states of the traffic system in the future,while the control parameters are optimized by a genetic algorithm.Then,the optimal control parameters are fed into a microscopic traffic simulation model to improve the overall traffic efficiency of the road network through a hierarchical control strategy.Also,the current state of the microscopic traffic simulation model is sent back into the regional boundary control model.Second,the microscopic signal timings at intersections are obtained based on deep reinforcement learning.To describe the vehicle dynamics at boundary intersections,a boundary intersection signal control model is developed to build an interactive environment for deep reinforcement learning agents.The number of waiting vehicles in the entrance lanes at the boundary intersections and the number of expected transfer vehicles（i.e.,the product of macroscopic control parameters and the transfer flow）are constructed as the state space.The given signal control strategy is used as the action space,and the reward function is designed according to the actual number of transfer vehicles and the expected one at the boundary intersections.The deep reinforcement learning algorithm achieves the actual number of vehicles transferred at the boundary intersection as close as possible to the expected value by maximizing the cumulative reward function.Subsequently,the deep Q-network and the proximal policy optimization algorithm are compared in multiple scenarios.The experiment results show that the proximal policy optimization algorithm performs better in terms of convergence effect and convergence speed.Finally,a virtual traffic network is built by the traffic simulation software SUMO.Its Tra CI interface is used to obtain MFD of the simulated road network as an evaluation tool.Subsequently,a signal control simulation model at boundary intersections is constructed as an interactive environment for the deep reinforcement learning agents.The agent switches signal phases dynamically by observing the vehicle states in the entrance lanes at the intersections,so as to control the traffic inflow and outflow at the boundary to ensure that the number of vehicles in the road network is as close to the optimal value as possible.Experiment results show that the regional congestion is alleviated to some extent.