Research On Decision-making And Control Based On Self-learning For Autonomous Driving

With the accelerated development of global urbanization,traffic safety and congestion,environmental pollution and other problems are becoming more and more serious.Autonomous driving technology is expected to solve these problems and has become an important research direction in modern automotive technology.In particular,the application of artificial intelligence-related technologies in the field of autonomous driving makes it possible to solve large-scale and complex urban scenarios for autonomous vehicle navigation.At present,most autonomous vehicles can only make decisions,planning and control in relatively closed scenes with high-precision maps.These methods are based on pre-set rules,and when the scene perceived by the vehicle is the same as the set scene,the behavior is determined according to the defined rules.Due to the complexity of autonomous driving scenarios,pre-set rules often cannot cover all scenarios,resulting in autonomous vehicle control failures and even fatal consequences.The self-learning ability of vehicles driven by data and artificial intelligence algorithms is expected to cope with complex autonomous driving scenarios.Autonomous vehicles need to perceive complex environments to make decisions and control the vehicle to reach the next environmental state.And vehicle achieves self learning in the continuous interaction with the environment to optimize the decisionmaking control scheme.This process fits the working paradigm of Reinforcement Learning（RL）,which models continuous decision-making problems as Markov processes and finds the optimal solution by solving the Bellman equation.However,due to the high computational complexity of reinforcement learning,it cannot solve the problem of high-dimensional（continuous）state space and behavior space.Deep Learning（DL）has strong perception ability and nonlinear function fitting ability,which provides the possibility to solve high-dimensional problems.Deep Reinforcement Learning（DRL）combines the powerful perception ability of deep learning with the reasoning ability of reinforcement learning,and has achieved remarkable results in the fields of mobile robots,robotic arms,and image-based games.Imitation Learning（IL）is another autonomous learning method.The agent learns policies through expert experience and generalizes to new scenarios,which makes up for the slow learning speed of DRL.Based on the joint fund "Research on Key Technologies and Platforms of Autonomous Driving Driven by Collaborative Intelligence",which was applied for by the Shenzhen Institute of Advanced Technology of the Chinese Academy of Sciences and the University of Macau,this paper conducts an in-depth study on the application of self learning in autonomous driving decision-making and control.The main work and innovations are as follows:（1）Aiming at the contradiction between the "trial and error" of traditional deep reinforcement learning and the safety requirements of autonomous vehicles,this paper proposes a deep reinforcement learning lane changing and overtaking framework based on dynamic minimum safe distance constraints.A reward function that comprehensively considers safety,efficiency and comfort is introduced to improve the performance of the lane changing strategy.The deep reinforcement learning framework based on the dynamic minimum safe distance constraint ensures that the vehicle is safety during training and testing.Using the Double DQN algorithm,the lane changing framework is instantiated,and it is trained and tested on the decision-making simulation platform Highway-env.The experimental results show that the lane changing strategy trained by the framework proposed in this paper improves the efficiency and comfort of overtaking under the premise of ensuring the safety of the vehicle.（2）Aiming at the problems of complex modeling and difficult parameter adjustment of traditional vehicle following control methods,this paper proposes a deep reinforcement learning control method based on multi-frame RGB images.A proximal policy optimization algorithm is used to control the acceleration and deceleration of the continuous behavior space.In order to solve the problem of slow learning caused by continuous behavior space,this paper adds expert experience to guide learning based on the experience generated by deep reinforcement learning and environment interaction.Environmental information is provided using structured and unstructured state spaces,respectively.The image information is input to the decision-making network as the environmental perception data,which is in line with the way of human driving perception and decision-making.To verify the impact of reward function on the optimization of deep reinforcement learning strategies,we use four different reward function training strategies.The experimental results show that,compared with the traditional car-following control method,the data-driven deep reinforcement learning method has strong generalization ability and high stability.（3）Aiming at the problem of autonomous driving navigation in large-scale and complex urban scenes,the imitation learning framework based on yaw angle guidance proposed in this paper realizes the end-to-end navigation of autonomous driving in urban scenes,and has a certain degree of interpretability.Imitation learning trains neural networks by collecting expert experience（observing pairs of actions）and generalizing to other scenarios.The traditional end-to-end method cannot solve the road selection in urban scenarios,that is,intersections and T-junctions.The conditional branch network uses the passenger’s instruction as the branch selection flag,and optimizes different branch network parameters.But this method reduces the efficiency of data usage.This paper proposes a method of yaw angle guidance,which effectively improves the efficiency of data use.The Attention mechanism is introduced into the perception module,and the Attention heatmap visualizes the key areas that the neural network pays attention to.It provides the basis for the analysis of the causal relationship between the scene and the decision.To sum up,the research on autonomous driving decision-making and control based on self learning makes up for the shortcomings of traditional decision-making and control methods based on rules and models that cannot handle large-scale and complex scenarios.It provides theoretical guidance for the application of artificial intelligence methods.