| In recent years,machine learning as an important method to achieve artificial intelligence,has attracted the attention of a wide range of researchers around the world.Reinforcement learning(RL),as an important branch of machine learning,emphasizes that agents learn in the process of interacting with the environment and use evaluative feedback signals to optimize decision making.Off-policy RL methods are theoretically provable and sample efficient,which are the most widely used.However,off-policy RL methods adopt greedy updating mechanism,which will lead to Q-value overestimation in the learning process,and thus affect the learning of optimal policy.On the other hand,in order to apply RL to the real world,safety considerations are indispensable.Directly allowing an agent to explore the environment with an unsafely constrained policy in the real world is quite dangerous.Thus,this thesis focuses on how to solve the problem of overestimation in the value function update process,as well as the safe exploration problems when applying RL methods to real world.The main work includes:(1)A new Alternated Greedy-step Update(AGU)framework that consists of two independent Q-value estimators is proposed to address the Q-value overestimation problem in Greedy-step Q-learning(GQL).In the proposed AGU framework,one estimator is to determine the time step that can maximize the estimated n-step return and the other one is to update the prior estimator using the target value calculated on the basis of the determined time step.The convergence and effectiveness of AGU framework is proved in theoretical.In addition,an Alternated Greedy-step Deep Deterministic Policy Gradient(AGDPG)that can be applied to continuous-action tasks is proposed by combining the AGU framework with Deep Deterministic Policy Gradient(DDPG).Experiments on Mu Jo Co and Atari platform highlights the superior performance of AGU framework and AGDPG.(2)To solve the safe exploration problem in real world,a new safe reinforcement learning algorithm named Twin Delayed Deep Deterministic Policy Gradient based on Approximate Safe Action(TD3-ASA)is proposed.In TD3-ASA,the action output by the current policy in the exploration process is modified to obtain an approximate safe action,and then the approximate safe action is used to learn a safe policy at deployment.Compared to previous algorithms,TD3-ASA has the following advantages: TD3-ASA is sample efficient and does not need any prior knowledge;TD3-ASA can ensure the safety both during training and at deployment;TD3-ASA provides an adjustable safety correction factor,which can be used to make a tradeoff between exploration and safety.Experimental results on Meta Drive and Speed Limit autonomous driving test platform show that: TD3-ASA effectively improves safety during exploration over current stateof-the-art RL methods,achieving a sufficiently high success rate and a sufficiently low deployment risk. |
