Deep Reinforcement Learning Based Decision-making And Control Method For Connected And Automated Vehicles

With the increasingly serious contradiction between the drastically increasing number of vehicles and the road capacity in the transportation system,the Vehicle Infrastructure Cooperative System(VICS)has become a strategic area for the development of the national transportation industry.Taking advantage of both VICS and autonomous driving technology,Connected and Automated Vehicle(CAV)is quickly becoming one of the transformative solutions to our current transportation problems.For CAV technology,the intelligent decision-making and control method under heterogeneous traffic environment has always been the opening challenge.At present,most of the state-of-art decision-making and control methods of CAV are based on mathematical models,which are limited by the uncertainty of the traffic environment.Other data-driven decision-making and control methods are limited by the ideal assumptions about traffic environments.Aiming at the specialty of the vehicle and complexity of the traffic environment,this paper proposed two deep reinforcement learning-based decision-making and control methods for CAV under the VICS-based heterogeneous traffic environments,including freeway and signalized intersections.Specifically,the research contents are showing as follows:1)In terms of the decision-making and control methods of CAV,this paper analyzes the state-of-art international and domestic achievements and briefly introduces the CAV-based VICS,typical traffic scenarios and the deep reinforcement learning theory.Then,we analyze the conventional and data-driven decision-making and control methods for CAV under freeway and signalized intersection environments.2)This paper proposes the VICS-based heterogeneous traffic environment model and the ego-vehicle model under freeway and signalized intersection environment.In addition,for each traffic environment,we propose the driving strategies,observation space,action space and reward function on the basis of reinforcement learning.3)In terms of the multi-source sensors and heterogeneous traffic environments,an interval-sampling-based data preprocessing method is proposed in this paper.For the freeway environments,a deep neural network based on parallel CNNs and Dueling Deep Q Network is proposed.On the other hand,for the signalized intersection environment,we propose a hybrid reinforcement learning framework which enables CAV to intelligently interact with the heterogeneous traffic.4)For the simulation experiments of the proposed methods,this paper studies the structure of simulator for reinforcement learning based algorithm.Then,we design and develop two simulators including freeway and signalized intersection under VICS-based heterogeneous traffic.Finally,the training and testing results are analyzed.The research results demonstrate that under freeway environment,the proposed decision-making and control model achieves 0.84% to 5.28% higher score according to penetration rate of connected vehicles,comparing with other baselines.On the other hand,for the signalized intersection environment,the proposed hybrid reinforcement learning model can save 1.13%-7.58% travel time and reduce energy consumption by 12.25%-56.05% when comparing with several baselines.Additionally,the proposed models could interact with the heterogeneous traffic environment more smoothly and have higher driving stability.