Research On On-Ramp Lane-Changing Decision Based On Deep Reinforcement Learning

China’s highway mileage has exceeded 140,000 kilometers,ranking first in the world,but at the same time the rate of traffic accidents is also high.The traffic environment in the confluence area is complex and prone to accidents,which is a major difficulty in driving highway vehicles.The decision-making of lane change in the confluence area is of great significance for the early realization of automatic driving above L3 level,alleviating traffic congestion,reducing the incidence of traffic accidents and improving the ecological environment of road traffic.Although the rule-based decision algorithm can ensure the real-time decision-making,it is easy to collapse when it encounters driving scenarios outside the rule base,and it cannot cope with the changes and uncertainties of the traffic environment.As a milestone breakthrough in the field of reinforcement learning,deep reinforcement learning can cope with the traffic environment with a lot of dynamic and uncertain information,and make reasonable decisions successfully in a complex environment.The author chooses deep reinforcement learning as the merge decision-making algorithm to study the automatic driving decision on the merge of highway ramps,and the problems of simplifying the interference car movement mode and lacking consideration of the impact on other traffic participants in the existing merge decision-making research are researched,the main research contents are as follows:(1)Research on the decision-making model of environmental vehicles autonomous lane change.Using Intelligent Driver Model to model longitudinal car following decisions,and using Overall Braking Induced by Lane Change Model to model lateral lane change decisions,so that the environmental vehicles have an independent driving target and unlimited mobility(Acceleration,deceleration,change lanes,etc.)behavior,and randomly select the starting position and target speed within a certain range,which can have a positive interaction with the host vehicle and restore the real lane-changing decision scenario to the greatest extent.(2)Build a virtual training environment and design the bottom controller.According to China’s transportation-related laws and regulations,the basic ramp merge area(including the form of acceleration lane,the length of lane,etc.)was designed,and the road passing rules were determined.Weighing experiment needs and actual conditions,modeling vehicle kinematics and adopting longitudinal control and lateral control methods based on PID control.(3)Research on the algorithm of lane change decision.Based on deep reinforcement learning theory and Markov decision process modeling of lane-changing behavior,a basic deep Q network decision model and a more advanced dueling deep Q network decision model are established.Considering the impact of lane-changing behavior on other traffic participants,two new reward functions are proposed for keeping the right lane and changing lanes for the speed of the following vehicle.The simulation training of the two decision-making networks shows that the dueling deep Q network decision-making model has a higher lane-changing success rate than the basic deep-Q network decision-making model,achieving a lane-changing success rate of more than 80% in all speed intervals;In the verification of the effectiveness of the new reward function,the decision network with the new reward function has a significant improvement in the two indicators of the impact of the following vehicle speed and the number of lane changes;finally,the trained dueling deep Q network is tested in three aspects,in the test of automatic vehicle speed change,all achieved a change rate of more than 90%;in the test of changes in the number of environmental vehicles,the success rate of decision-making network in different vehicle environments is more than95%,and in 6 vehicles environment,the success rate of vehicles exceeds 99%;in tests of changes in the driving style of environmental vehicles,the success rate of lane changes exceeds 98%.Through the above simulations and tests,it has fully demonstrated the excellent performance of the decision-making network based on deep reinforcement learning to handle lane change decision-making problems and the strong adaptability to deal with uncertain environments.