Research On Ship Path Planning Method Based On Deep Reinforcement Learning

With the rapid development of artificial intelligence and other high-tech,it is of great theoretical and practical significance to explore more intelligent ship route planning methods for the development of safe navigation and intelligent shipping industry.Compared with the traditional ship route planning method,the ship route planning method based on deep reinforcement learning seeks the optimal strategy of the ship through continuous "trial and error" learning,which makes it easier for the ship to find a safe and effective route in the complex environment.However,in the current ship route planning methods based on deep reinforcement learning,the characteristics of ship navigation are seldom combined;collision avoidance rules are seldom considered in the planning of collision avoidance between ships,which is inconsistent with the actual navigation situation;moreover,when dealing with the situation of multiple ships,most of the current ship route planning methods are based on the collision risk degree of the ship and other ships to avoid and guide one by one As a result,the navigation route of this ship is greatly affected by other ships,and the planned route is not reasonable;in addition,because the surrounding state of the ship is constantly changing during the navigation process,this text takes into account the global route planning(safety and economy)and local route planning(safety and timeliness)based on the navigation state of the ship("single ship state" or "multiple ship state")In order to meet the requirements of this text,the method of ship route planning based on deep reinforcement learning is explored.The main research work and achievements are as follows:1)Research on collision avoidance strategy in ship route planning based on deep reinforcement learningBased on the analysis of the characteristics of ship navigation,the risk assessment method of ship collision is designed based on the field of inland ships.At the same time,the ship navigation status is divided into "single ship status" or "multiple ship status";the rules of collision avoidance for inland ships are studied,and the collision avoidance strategies under three situations are designed;the coincidence of ship path planning and depth reinforcement learning are analyzed,and the ship collision risk assessment method is proposed In different states of navigation,the collision avoidance strategy is integrated when the path is planned based on different depth reinforcement learning algorithm.2)Research on route planning method of single ship for safety and economyWhen the ship is in the state of "single ship",the safety and economy are taken as the goal of ship path planning,the risk of ship collision is combined with sac,which is a deep reinforcement learning algorithm of single agent,and the reward function of sac is redesigned,so that the ship can learn to adapt to the environment,effectively avoid static obstacles and ensure safety when planning the route;the sac algorithm is applied to ship path When planning,the exploration strategy is not limited by safety.It is improved to design a dual experience sample pool,distinguish the types of samples generated in the training process,and build a dual critical network to train different types of samples,reduce the number of times that the ship once again falls into a dangerous state,thus limiting the unrestricted exploration of the ship in the path planning training,and ensuring the safety of the ship High algorithm training speed;design a single ship path planning method based on the improved sac algorithm,and realize the economy of ship path planning under the premise of safety.3)Research on ship path planning method for multi ship collision avoidanceWhen the ship is in the state of "multiple ships",taking safety and timeliness as the goal of ship path planning,combining ship collision risk with multi-agent depth reinforcement learning algorithm maac,redesigning the reward function of maac algorithm,making the ship learn to adapt to the environment to achieve the safety of the path;according to the characteristics of ship navigation,improving the Q-value calculation method of maac algorithm,proposing It improves the efficiency of algorithm planning;aiming at the problem that the random sampling of maac algorithm leads to low learning efficiency and slow convergence speed in the training process,an empirical sample extraction method based on priority is proposed,which takes priority as the extraction probability to obtain learning samples for training,so as to speed up the convergence speed of the algorithm;the designed ship collision avoidance strategy is integrated into the path planning decision-making to guarantee the multi ship relationship Based on this,a multi ship cooperative collision avoidance path planning method based on improved maac is proposed to plan a safe collision free route for each ship,and the planned route meets the actual navigation requirements.4)Experiments and AnalysisThe safety,timeliness and economy are selected as the evaluation indexes to verify and analyze the intelligent ship route planning method and its improvement strategy designed in this text.At the same time,it is compared with other mainstream ship route planning methods by experiments.The results show that the intelligent ship route planning method designed in this text is correct and effective,meets the actual navigation needs of ships,and has practicability.