Research On Underwater Robot Navigation Algorithm Based On Deep Reinforcement Learning

Due to the continuous depletion of land resources,many countries have turned their attention to the oceans with abundant mineral resources and fishery resources.Intelligent underwater robots can autonomously implement the exploitation of marine resources in the underwater environment to avoid the safety risks caused by artificial underwater operations.The autonomous navigation capability of underwater robots is an important prerequisite for their successful underwater operations.In order to solve the problems that the conventional autonomous navigation algorithm has too large amount of calculations in a complex underwater environment to realize real-time navigation,and reinforcement learning will encounter dimensional disasters and it takes a lot of time to train and learn,this topic proposes an DDQN algorithm to study the autonomous navigation of underwater robots in unknown environments.The algorithm gives underwater robots the ability to learn autonomously,improves the adaptability of robots working in different environments,and solves the bottleneck problem of traditional autonomous navigation algorithms.,To achieve autonomous navigation of underwater robots in a mapless information environment.The underwater robot navigation task can be decomposed into two sub-tasks: local obstacle avoidance and global navigation.Therefore,this paper proposes a modular neural network structure,correspondingd to the local obstacle avoidance neural network module and the global navigation neural network navigation module.The local obstacle avoidance neural network module is mainly used to guide the underwater robot away from obstacles;the global navigation neural network module is mainly used to guide the underwater robot to the shortest path to the target position.Aiming at the problem of sparse rewards for underwater robots during training,a new continuous combined reward function is designed to improve the convergence speed of the algorithm.The reward value for the conventional DDQN algorithm can only affect the state-action Q value of adjacent states.This paper proposes a multi-step mechanism DDQN algorithm(Mu Lti-step DDQN,MS-DDQN).This method realizes the extension of the influence of the reward value to the state-action value of the next few states,which is equivalent to improving the ability of the underwater robot to sense the future state information,so that the underwater robot has a stronger ability to navigate and avoid obstacles.In order to speed up the convergence of deep neural networks and enable underwater robots to learn an optimal path navigation strategy,a new continuous combination reward function is proposed.The direction reward and danger reward are added on the basis of the original reward function.These two rewards guide the robot away from obstacles and toward the target position.In this paper,geometric methods are used to describe the environmental information of obstacles,and the obstacles are expanded accordingly,and then the underwater robot can be treated as a particle.This paper use Python language and Pygame library to construct training simulation environment model and simulated underwater robot model.And adopt deep learning framework Tesnorflow to build a modular neural network structure.Finally,the MS-DDQN algorithm is used to train the autonomous navigation model of the underwater robot in the simulated underwater environment.By testing underwater robots in different test environments and comparing and analyzing the experimental results,it is proved that the proposed MS-DDQN algorithm has higher learning ability and stronger generalization ability than traditional DDQN.Underwater robots based on MS-DDQN algorithm can achieve autonomous navigation in different unknown complex environments without retraining for new environments.