Human-like Adaptive Cruise Control Algorithm Design Based On Deep Reinforcement Learning

Adaptive Cruise Control System(ACC)is also known as active cruise control system.As a kind of advanced driving assistance function,ACC monitors the road traffic environment in front of the car through sensors such as on-board radar.After comparing with the information of the car,ACC controls the longitudinal speed of the car,so as to maintain a proper safe distance between the car and the car in front.In order to provide passengers with a comfortable driving experience and enable the surrounding drivers to better understand the behavior of the car,the driving style of ACC should be close to that of human drivers,so it is necessary to design an ACC algorithm that can imitate the driving style of human,namely the human-like ACC algorithm.At present,ACC algorithm is mainly divided into rule-based ACC algorithm and learningbased ACC algorithm.In the design of ACC algorithm based on learning,most researchers use the single-layer deep reinforcement learning algorithm to build ACC and the expert estimation method to design the reward function.Since the single-layer reinforcement learning algorithm does not conform to the human "decision-execution" behavior pattern,and the reward function designed by the expert estimation method does not like the reward function based on the real driving data,it is difficult to design the personalized ACC algorithm based on this method.In order to solve these problems,a new ACC algorithm structure is designed in this paper.The hierarchical reinforcement learning algorithm is used as the basic framework,and the humanlike reward function is learned from the real following driving data based on the reverse reinforcement learning algorithm.Finally,the human-like ACC algorithm is realized.The main contents of this paper are as follows:A new adaptive cruise control algorithm is designed based on hierarchical reinforcement learning.This article to imitate human behavior patterns of “decision – execution”,hierarchical reinforcement learning algorithm as the basic framework,in Deep Q Network algorithm(DQN)as the top decision-making algorithm and the underlying implementation algorithm to build the adaptive cruise control algorithm based on safety,comfort and preliminary design reward function.The learning speed and actual performance of adaptive cruise algorithm are optimized by improving DQN algorithm.Aiming at the low training efficiency of initial DQN algorithm,this paper improves the neural network structure,memory access and algorithm training flow of DQN algorithm,and forms a new coupling DQN algorithm.The simulation results show that the coupling DQN algorithm has a significant improvement in learning efficiency and actual performance compared with the original DQN algorithm.Learning human-like driving strategies from real driving data based on reverse reinforcement learning.This article collects some skilled Driver in the city of open road driving real driving data as the test sample,and based on the maximum entropy reverse reinforcement learning theory to design the human-like adaptive cruise control algorithm reward function learning algorithm,feature mapping,state transition probability and expected characteristics calculation and reward function coefficient and update process,make the algorithm to track the expectations of the eigenvalue is closer to the expectations of human driving data characteristic value,completed the algorithm of adaptive cruise control strategy of human-like learning driving.Comparison of training and human-like effects of adaptive cruise control algorithm.In this paper,training conditions are designed based on real driving data samples,and pre-training of the underlying execution algorithm under simple and complex conditions is carried out respectively.Then,the adaptive cruise control algorithm is trained as a whole under the same conditions.The test results show that the expected characteristic values of safety,comfort and followability of the human-like reward function designed in this paper are closer to the expected characteristic values of human driving data,that is,the human-like adaptive cruise control algorithm is designed.