Research On Key Technologies Of Autonomous Ship Collision Avoidance Decision-making Based On Deep Reinforcement Learning

In recent years,with the development of artificial intelligence,transportation vehicles such as cars and boats have gradually achieved autonomous control,and unmanned systems like drones and robots have been maturely applied.Autonomous ships have received much attention due to their potential to reduce human errors,ensure navigation safety,and improve transportation efficiency.However,the navigation technology of autonomous ships faces significant challenges in complex sailing environments,especially in collision avoidance decision-making.Unlike other transportation vehicles such as unmanned cars,the maritime traffic environment is complex and variable,and the unstructured sailing environment presents numerous uncertainties in autonomous ship collision avoidance decision-making.Conventional cargo ships still heavily rely on crew members for situation assessment and planning avoidance strategies,but traditional collision avoidance decision-making techniques for autonomous ships have significant drawbacks in dealing with scenario diversity and adaptability,and collision avoidance decision-making in busy waters remains to be solved.Deep reinforcement learning(DRL),as a representative algorithm of interactive learning methods,embodies the characteristics of learning through interaction with the environment,which is in line with the human evolutionary process of adapting to environments or new things.Therefore,this thesis systematically introduces DRL algorithms and conducts research on the key scientific issues of intelligent collision avoidance decision-making for autonomous ships in busy waters.From the basic modeling of supervised collision avoidance decision-making in DRL to gradually advancing the application of DRL to solve practical engineering problems,this thesis explores new approaches for human-like collision avoidance decision-making for autonomous ships.The main research work of this thesis includes:(1)To address the scientific issues of multi-constraint Markov decision processes involved in collision avoidance decision-making in busy waters,a multi-constraint reinforcement learning-based autonomous ship collision avoidance decision-making algorithm is proposed.Firstly,based on the navigation characteristics of autonomous ships,an abstract constraint model for collision avoidance decision-making of autonomous ships in busy waters is constructed,covering aspects such as collision risk,ship maneuverability,and International Regulations for Preventing Collisions at Sea(COLREGs).Then,based on this foundation,a collision avoidance decision-making model based on multi-constraint reinforcement learning is constructed from perspectives such as obstacle detection,steering-dominated avoidance action space,reward functions,and action selection strategies.Finally,the applicability and effectiveness of the model and algorithm are verified through simulation experiments,laying the foundation for an intelligent collision avoidance decision-making algorithm in busy waters.(2)In view of the characteristics of environmental uncertainty in busy waters and the highdimensional navigation situation parameter space of ships,an efficient reinforcement learning collision avoidance decision-making algorithm using experience reuse is proposed.Firstly,the current status of traditional learning-based collision avoidance decision-making algorithms is analyzed,and the problems of low sample utilization and slow iteration speed in uncertain environments in busy waters are described.Based on this,considering constraints such as COLREGs,a mixed reward function model combining external reward signals and internal incentive signals,including position rewards,speed rewards,target rewards,and shaping rewards,is constructed to obtain the optimal interactive learning scheme.Then,by mining the hidden features of historical training data,the accumulated learning experience is reused in the utilization phase and for updating the initial values of the value function update table.Finally,multiple sets of simulation experiments are designed to verify the effectiveness of the model and algorithm.This method effectively improves the search efficiency for busy water environments and the learning efficiency for high-dimensional navigation situation parameters.(3)To address the technical challenges of a large amount of nonlinearity and highdimensional continuous state space brought by the high-dynamic interactions between ships in busy waters,a collision avoidance decision-making algorithm combining approximate representation and deep reinforcement learning is proposed.In continuous state space problems,it is difficult for algorithms to accurately search for each state and store each state value function or state-action value function.Therefore,an approximate representation mechanism based on coarse coding technology is proposed,which can realize the approximate search and storage of value functions and policies under prior knowledge constraints.This improves the effectiveness of decision-making while reducing the cost of algorithm-environment interaction search.Finally,a gradient descent value function approximation solution method is designed.Through Python simulation experiments,the results show that the algorithm can effectively solve the collision avoidance decision-making task for autonomous ships in busy waters,avoiding the catastrophe of massive nonlinearity,high-dimensional continuous environment online search,and solving the technical challenges of continuous state space in the highly dynamic interaction process of ships in busy waters.(4)To further enhance the safety and reliability of intelligent collision avoidance for autonomous ships in busy waters,a safe hierarchical reinforcement learning collision avoidance decision-making algorithm and reliability assessment method based on risk assessment are proposed.Firstly,aiming at the multi-ship encounter problem in large-scale,traffic-intensive real navigation,the UG theory is introduced to construct a multi-ship joint risk assessment model.Secondly,by introducing collision risk factors into the reward function,objective function,and learning process,a collision avoidance decision-making algorithm based on safety-layered reinforcement learning is designed,and an innovative reliability assessment network is designed for this algorithm to optimize its engineering applicability.Finally,taking Zhoushan Island waters as an example,a simulation experiment that integrates sequential collision avoidance and joint dynamic obstacle avoidance is designed,and the algorithm results are analyzed from multiple perspectives such as safety,efficiency,and reliability.This method contributes to applying deep reinforcement learning to the safe collision avoidance decisionmaking and autonomous navigation of real ships.