Research On Multi-agent Deep Reinforcement Learning For Large-scale Traffic Signal Control

As the amount of vehicle in cities increases,the load on roads becomes larger,and traffic congestion in urban transportation system become more and more serious.The capacity of road is mainly affected by intersections,whose capacity is limited by the existing signal control law,indicating large room for optimization.Therefore,it is of important research value that the collaborative optimization for large-scale intersection traffic patterns in urban systems.In this paper,we investigate the large-scale intersection signal control based on multi-agent deep reinforcement learning theory.This study aims at the application of multi-intelligent reinforcement learning algorithms on the problem of large-scale signal control in intersections,including following aspects: Firstly,the coupling mechanism between signal decision and traffic flow is investigated,with a concept of signal control region proposed.The large-scale signal control problem is modeled as Markov decision process,where the signal at each intersection is selected as an agent,and then the action,state and reward function of each agent are defined based on the mentioned signal control region.Secondly,the agent learned in the frame of reinforcement learning based on value function and policy gradient,respectively.The most state-of-the-art learning algorithms under each frame are selected and utilized,which are DDQN(Double deep Q-learning)and SAC(Soft-Actor-Critic).Corresponding learning frames for multi-agent cooperative decision-making problem are designed,where the joint-state and joint-reward of multi agents are established.Next,a novel idea called greedy decision order for multi-intelligent reinforcement learning methods is proposed and applied to the previously mentioned reinforcement learning frames.Then the environment where large-scale signal decision-making interacts with traffic flow is established by SUMO,a traffic flow simulation software,and the training framework of deep reinforcement learning algorithm is achieved by Tensorflow in python.The agent is finally jointly trained with SUMO.Finally,the deep neural network decision model is obtained based on the training of the proposed algorithm,and the performance and convergence speed of the model are verified under the large-scale traffic intersection signal control problem.Also,it is compared with the traditional multi-agent reinforcement learning algorithm.The multi-intelligent reinforcement learning algorithm with greedy decision order proposed in this study makes improvement based on the traditional multi-intelligent algorithm and tries to achieve the collaborative decision-making between agents.The final trained decision model can effectively improve the traffic efficiency of intersections.Compared with traditional optimization algorithms,the trained decision model has the capacity of generalization;compared with single-agent deep reinforcement learning algorithms,it can realize the collaborative optimization between different intersections,and has better optimization performance in large-scale problems;compared with traditional multi-intelligent body algorithms,it simplifies the dimensionality and complexity of the problem and has better convergence speed.In conclusion,the work of this paper improves the multi-agent deep reinforcement learning algorithm on the one hand,and on the other hand provides an efficient solution for the large-scale signal control problem.