Research On Ship Collision Avoidance Decision Based On Deep Reinforcement Learning

With the rapid development of artificial intelligence,Internet of Things,big data and other technologies,ship intelligence has become an inevitable trend in the future development of shipbuilding and shipping industry.Moreover,with the increasing frequency of maritime traffic activities,the frequency of ship collision accidents caused by human factors remains high,which seriously threatens the safety of life,property and water environment.Therefore,improving the intelligence level of ship collision avoidance decision-making has important practical significance for reducing or avoiding ship collision accidents and improving the safety level of water environment.In order to meet the development needs of intelligent shipping and autonomous navigation of intelligent ships,realize safe and autonomous collision avoidance among ships,and solve the problem of ship collision avoidance decision-making methods based on traditional deep reinforcement learning,such as weak generalization ability,weak connection with collision risk and weak robustness in complex environment.The paper studies ship collision avoidance theory,ship collision risk quantification and ship autonomous collision avoidance based on deep reinforcement learning.A ship autonomous collision avoidance method based on Twin Delayed Deep Deterministic Policy Gradient(TD3)is proposed,which provides theoretical and technical support for ship intelligence and autonomous navigation.The main research contents are as follows:(1)The ship collision avoidance decision theory based on reinforcement learning.By analyzing the traditional ship avoidance process,the division of encounter situations,collision avoidance strategies and avoidance action requirements,the basic principles of ship collision avoidance are summarized,and a ship collision risk detection method based on safety distance is proposed.The paper analyzes the fit between reinforcement learning and autonomous collision avoidance,and the solution process of the optimal collision avoidance strategy.Aiming at the defect of traditional reinforcement learning,which is difficult to solve the problem of complex ship autonomous collision avoidance decision-making,the deep neural network technology is used to propose a deep reinforcement learning-based ship autonomous collision avoidance algorithm framework.(2)The collision risk model based on the quaternion ship domain.In view of the problems that the existing collision risk quantification methods have insufficient consideration of risk factors,cannot accurately reflect the degree of danger between ships,and are not suitable for autonomous collision avoidance scenarios of ships.The safety distance boundary is determined according to the quaternion ship domain.The collision risk by using the collision risk detection circle and collision risk detection line to identify.The paper takes into account the relevant factors that affect the collision risk of the ship’s autonomous navigation to form a risk evaluation index.Using the fuzzy theory,a fuzzy evaluation model of the ship collision risk based on the fourdimensional ship field is proposed,which can help the ship’s autonomous avoidance,which provides basic support for ship autonomous collision avoidance model.(3)The decision-making method for ship autonomous collision avoidance based on Twin Delayed Deep Deterministic Policy Gradient.Aiming at the shortcomings of traditional deep reinforcement learning algorithms in the application of ship autonomous collision avoidance,based on the TD3,a state space with continuous multi-time target ship information is constructed from a global perspective to enhance the robustness of the model.The continuous action space is designed according to the ship maneuverability,and the ship collision avoidance reward function is designed in combination with the collision risk model and the COLREGS.According to the ship collision avoidance state space,the ship autonomous collision avoidance network model including Long Short Term Memory(LSTM)network units is designed by using Actor-Critic structure.The network training is stabilized by “Clipped Double Qlearning”,“Delayed Policy Updates” and “Target Policy Smoothing”,so as to further enhance the robustness of the model.In order to solve the problem of poor generalization ability of autonomous collision avoidance model,a random scene training process of ship autonomous collision avoidance algorithm is proposed to realize the multi scene migration of collision avoidance model application.(4)Training and simulation verification of ship autonomous collision avoidance algorithm.Using the constructed random encounter scenario and training parameters,the ship autonomous collision avoidance algorithm is trained,and the autonomous collision avoidance network model is obtained.The model is simulated and verified in the encounter scenario of two ships and multi-ships respectively,and compared with the existing typical ship autonomous collision avoidance methods to verify the effectiveness and reliability of the proposed method.Based on the development needs of intelligent shipping and intelligent ship autonomous navigation,the paper studies the decision-making method of ship autonomous collision avoidance based on deep reinforcement learning theory.The research of this paper includes the key contents of ship collision avoidance principle,ship collision risk quantification and ship autonomous collision avoidance methods,which has important application value for ship intelligence and autonomous navigation.