Research And Implementation Of Highway Ramp Metering Algorithm And Simulation Platform Based On Deep Reinforcement Learning

Ramp Metering is a direct,effective and widely used highway traffic control measure by planning the timing and quantity of ramp traffic merging into the mainline through ramp signal,thereby alleviating traffic congestion and ensuring the operational efficiency of the mainline of highways.At present,the most widely used ramp metering methods are mostly based on traditional feedback control,but these methods have problems such as slow response speed and inaccurate control.In addition,methods based on nonlinear optimal control rely on the premise of a known and accurate system model,but modeling complex traffic systems is difficult and prone to errors,leading to significant deviations in control results.Control methods based on heuristic algorithms rely on empirical data and prior knowledge,resulting in unstable algorithms with poor adaptability.This paper aims to study an intelligent ramp metering algorithm based on deep reinforcement learning,which has the characteristics of real-time response,independence from system modeling,and strong robustness.With the construction of an intelligent ramp metering algorithm as the core,the main research contents of this paper are as follows.(1)The pre-task of the ramp metering algorithm is studied,that is,the traffic flow parameter extraction algorithm based on video data.The performance of the ramp metering algorithm depends on the accurate perception of the traffic environment.In view of the current situation of domestic highway infrastructure and the rapid development of computer vision,installing a video image detection system is the most cost-effective traffic detection method.Therefore,building a traffic flow detection parameter extraction algorithm based on detection and tracking.Firstly,road traffic video datasets is constructed.Vehicle location information is extracted based on the Faster R-CNN target detection model.The Deep SORT multi-target tracking algorithm combines the vehicle detection results to obtain the vehicle trajectory.Finally,according to the tracking track information,the extraction method of traffic flow parameters is studied,and the key traffic parameters are analyzed and extracted,including flow,density and speed.The experimental results show that the adopted algorithms have good real-time performance and accuracy,which provides theoretical support for the environment perception module of the ramp metering algorithm.(2)Study the core task of this paper,namely,the intelligent ramp metering algorithm based on deep reinforcement learning,by constructing the DDQN-RTD ramp metering model based on real-time detection and response.For the ramp metering problem,a DDQN model based on value function approximation is established,and the traffic states are analyzed based on key traffic flow parameters,and then the environmental state representation is designed.State encodings are used as input to the control model.Abandon the concept of fixed control period and the way of periodically updating the detector in the traditional ramp metering method.Define the action space directly as red and green lights.And based on discrete time and unit step,real-time traffic parameter detection and update are performed.At the same time,the reinforcement learning agent gives real-time decision-making information within a unit step,and combines the phase conversion constraints of signal lights to obtain control actions.The ramp metering system reward mechanism is established based on part of the traffic data to measure the current action value.The co-simulation platform completes the model training.During training,the agent learns the optimal value function based on the trial-and-error mechanism,the experience playback mechanism,the establishment of the target network,and the time difference method in the real-time interaction with the environment,and finally obtains the optimal control strategy.(3)Build a Web-based ramp metering simulation platform and solve cross-platform communication problems based on the Eel framework.The simulation platform realizes static visualization tasks of simulation environment and simulation objects based on HTML.Create vehicles,roads,detectors,controllers and other traffic objects based on Java Script to simulate the real road environment and realize the dynamic operation of the simulation environment.The real-time communication between the Python control algorithm side and the Web simulation side is realized based on the Eel framework.Finally,a stable experimental environment is provided for cross-platform algorithm training and testing.(4)Verify the performance of the ramp metering algorithm based on the simulation platform.The performance of the proposed DDQN-RTD algorithm is compared with the DDQN algorithm based on the timing detector.The results show that the real-time update of the detector improves the accuracy of ramp metering and the balance between mainline performance and ramp performance.The performance of the DDQN-RTD algorithm is compared with the classic ALINEA algorithm and the algorithm based on regression and clustering(ML-C&R).The results show that the DDQN-RTD method can effectively improve the service level of the mainline during peak hours.Based on the uncontrolled situation,the DDQN-RTD method reduces the average travel time of the mainline by 65.87%,which is further reduced by 8.2% and 10.67% compared with the ALINEA method and the ML-C&R method.The DDQN-RTD method increases the throughput of the mainline by 78.23%,which is further improved by 12.79% and 15.3% compared with the ALINEA method and the ML-C&R method.In addition,the ramp throughput is similar to the comparison method,and the average travel time on the ramp is further shortened.