Application Research Of Multi-agent Reinforcement Learning In Multiple Target-ships Collision Avoidance Decision-making

With the development of global economy and the increase of transnational trade,waterway transportation has gradually become the most important mode of transportation.Therefore,the construction of water transport is included in the “14th Five-Year Plan 102 important projects”.However,this will undoubtedly increase the congestion of maritime traffic and lead to more maritime traffic accidents.When there are many ships near the transport ship,it is easy to make wrong decisions by relying on people to manipulate the ship.According to data,about 89-96 % of maritime collision accidents are man-made.In order to avoid this situation,automatic collision avoidance of ships has become one of the most important research issues in the field of marine engineering.The main research contents are as follows:(1)The collision avoidance decision-making of the ship is divided in detail according to the International Regulations for Preventing Collisions at Sea(COLREGs),and the relevant ship models are established.Firstly,according to the COLREGs,the ship collision avoidance situations are discussed in detail,and the complete ship collision avoidance process is given.The encounter situations between the own ship and the target ship are carefully divided.Next,it is necessary to establish a ship motion model that conforms to ship maneuverability and a ship safety field model.Finally,since the ship may encounter multiple target ships,a ship collision risk model is established to rank the collision risk priority of the target ship to ensure that collision avoidance is carried out in an orderly manner.(2)The basic principles and mathematical models of reinforcement learning are elaborated in detail.Aiming at the problem that reinforcement learning is easy to fall into local optimum due to the lack of information interaction in the training process when solving multi-ship collision avoidance,the QMIX algorithm in multi-agent reinforcement learning is proposed.The use of multiple QMIX algorithms can enhance the information interaction between ships and get better training results.In the training process of QMIX algorithm,the influence of environmental factors on ships is ignored due to the loss function.Therefore,the SQMIX algorithm is proposed to add the environmental information part on the basis of the original loss function,and the weight coefficient is set to prevent the global information from affecting the training too much,resulting in the inability to converge.(3)Due to the two tasks that need to be completed during the ship’s navigation: path tracking and collision avoidance,a reward function that can switch modes is set.When there is no collision risk,the path tracking mode is used,and when the ship invades the safety field,it switches to the collision avoidance mode.(4)Firstly,simulation is carried out for four classic two-ship encounter situations(head-on,port crossing,starboard crossing and overtaking).The convergence analysis shows that the overall convergence speed of SQMIX algorithm does not decrease after introducing environmental information.The collision avoidance route is analyzed,the results show that the SQMIX algorithm can make collision avoidance decisions that are in line with ship maneuverability and safety.Finally,in order to prove the scalability of the SQMIX algorithm,the multi-ship collision avoidance scenarios are set.The results show that the SQMIX algorithm can realize automatic collision avoidance of multi-target ships.This thesis introduces the collision avoidance theory and ship motion model.In order to calculate the collision avoidance priority of multi-target ship,a ship collision risk model is established.Then,aiming at the problem that QMIX algorithm ignores the update of environmental information,the environmental information loss function with weight coefficient is introduced to ensure the convergence speed of the algorithm and improve the convergence accuracy.Finally,the reward function of the switchable mode is set.When there is a collision risk,it will automatically switch to the collision avoidance reward function.When there is no collision risk,it will be in track keeping mode.