| As an important tool for exploring the ocean,Unmanned Underwater Vehicle(UUV)is widely used in seabed exploration,underwater mapping,marine surveying and other tasks.Path planning is one of the key technologies for UUV to perform tasks.In order to make UUV obtain the evaluation optimal path faster in path planning,this paper uses reinforcement learning to find and learn the optimal strategy to improve the ability of UUV to search the optimal path in an unknown environment.In addition,the exploration strategy and learning method are improved by measuring the uncertainty of action value and experience value of reinforcement learning to improve the performance of path planning algorithm.The specific research contents are as follows:Aiming at the problem that traditional Q learning will fall into dimension disaster in complex environment,this paper introduces DQN algorithm in UUV path planning,which makes UUV have larger action space,and can receive a lot of sonar information,so that the simulation environment is more realistic.According to the steering ability of UUV,the action space of UUV in reinforcement learning is designed,and the course of UUV,the distance between UUV and obstacles,and the navigation time are used as parameters to design the reward and punishment function of action.Aiming at the problem that the traditional DQN algorithm is easy to fall into local optimum and difficult to balance exploration and utilization when exploring the environment,this paper proposes a DQN algorithm based on action value uncertainty.The algorithm determines the exploration mode through neural network,and can evaluate the uncertainty of action value.The multi-step exploration of underwater environment is realized by measuring the real value of the action to select the action.In view of the over-estimation problem of DQN network,a mean-based Q estimation calculation method is proposed to realize the decoupling of action selection and estimation,and improve the accuracy of action value evaluation.The simulation results show that the improved DQN algorithm can solve the optimal strategy.Compared with the conventional DQN algorithm,it not only shortens the planning time,but also improves the reward value in the UUV path planning.Aiming at the low efficiency of reinforcement learning in the learning process,an empirical learning method based on empirical value uncertainty is proposed.The algorithm combines the value uncertainty of TD-error and experience,by increasing the sampling priority of high value samples and setting association parameters to ensure that all high value experience is fully learned by the network.The simulation results show that the proposed algorithm can accelerate the convergence speed of network model and improve the path planning ability of reinforcement learning algorithm in complex environment. |
