Research On Path Planning Of Underwater Unmanned Vehicle Based On Curriculum Double Deep Q Network

In the working environment of Unmanned Underwater Vehicle(UUV),there are a large number of interfering factors that will affect the safe navigation of UUV.Path planning is the key technology to support UUV to achieve autonomous navigation.It runs through the underwater navigation of UUV and is the basis for completing underwater tasks.In this paper,curriculum reinforcement learning method is applied to the path planning task of UUV,endowing agents with the ability of self-learning and autonomous decision-making,and improving the environmental adaptability of UUV.The main research contents and achievements of this paper include the following aspects:1.Simply using reinforcement learning to realize path planning must take a long time and may be impossible to converge.Aiming at its disadvantages of unstable training and long training time,a curriculum learning Double Deep Q Network(Double DQN)algorithm for UUV path planning was proposed.This algorithm integrates the experience playback pool technology,effectively breaks the correlation between training data,improves the utilization of data,and effectively reduces the training time.Double DQN can effectively eliminate the maximum deviation brought by Q learning and enhance the stability of training.2.Based on Markov decision theory,the global path planning design of UUV system is carried out.It includes setting up the UUV task structure based on global path planning,establishing the reward function based on the path planning task objective and the environment state and behavior action model required by global path planning of UUV.Gym platform was used to create a UUV global path planning simulation environment including underwater environment,interaction,visualization and other functions.3.Aiming at the low learning efficiency of Double DQN algorithm in complex planning tasks,Curriculum Learning training method was proposed to optimize Double DQN algorithm.During the training process,the weights of the samples were dynamically allocated.In the initial stage of the course,most samples are simple,while at the end of the course,the samples are more difficult.Until finally converging on complex mission objectives.The simulation results of path planning of underwater unmanned vehicle under static and dynamic obstacles and ocean current interference show that the curriculum training method can effectively accelerate the learning speed of Double DQN in complex planning tasks.There will be disadvantages such as unstable training and long training time using reinforcement learning,a curriculum Double Deep Q Network(Double DQN)algorithm for Unmanned Underwater Vehicle(UUV)path planning is proposed.It incorporates experience playback pool technology,effectively shortening the training time,eliminating the maximum deviation caused by Q learning.Meanwhile,the curriculum-based learning is designed to improve convergence rate for the DDQN algorithm.By UUV’s global path planning simulations under static and dynamic environments,the effectiveness of the curriculum Double Deep Q Network algorithm is verified.