Human-like Longitudinal Velocity Control Based On Continuous Reinforcement Learning

With the development of intelligent transportation systems,intelligent driving technologies that can drive vehicles autonomously with little human interventions have attracted growing attention from both researchers and the public.However,when multiple factors,such as passenger comfort and driving smoothness,are considered,such kind of intelligent driving systems will suffer from low efficiency because of their rigid planning and control modules.On the other hand,experienced human drivers can deal well with multiple factors in the real traffic environment without a complex planning and control system.With this in mind,this paper modifies the framework of the traditional intelligent driving system,aims to develop a learning-based system that can learn from human drivers and realize the human-like control by connecting the learning system with the traditional control system.The whole system contains two main parts: a learning module for learning human driving strategies and a traditional PID control module which can convert the strategies learned to specific control actions for the throttle and brake of vehicles.The learning module is based on reinforcement learning(RL),which is an online learning method with various applications for real-time learning.To be more specific,Neural Q-learning(NQL)is selected to deal with the problem of learning expected longitudinal velocity from human manipulations,which is considered as a continuous control problem.Compared with the basic Q-learning algorithm with discrete state and action spaces,NQL uses an artificial neural network(ANN)to approximate the Q-function and avoid the curse of dimensionality,thus is suitable for continuous problems.Experiments based on simulation and real vehicle have been done to test the performance of the proposed system.A driving simulator based on PreScan is used to collect the driving data from human drivers.The experimental results are as follows:(1)the fixdistance simulation experiments show that NQL method can be used to solve the longitudinal-velocity control problem,(2)the simulation experiments in different scenarios show that the learning system can duplicate human driving strategies with acceptable errors,but perform worse in scenarios with changeable velocity.Moreover,compared with the traditional adaptive cruise control,the proposed system can provide better driving comfort and smoothness in the dynamic situation,(3)because the real vehicle data is more complex than the simulation data,the performance of learning systems declined in the real vehicle test.