Collision Avoidance Decision Of Maritime Autonomous Surface Ships Based On Deep Reinforcement Learning

With the rapid development of shipping economy and technology level,the level of navigation intelligence has been rising.In order to ensure the safety of shipping,the problem of intelligent collision avoidance is one of the hot spots of intelligent ship research.As the development of Maritime Autonomous Surface Ships(MASS)is still in its early stage,there will be a long period of coexistence between manned ships and MASS in the future,and the collision avoidance problem of both will become an urgent problem to be solved.Based on this,this thesis takes complex waters as the research area and uses “Collision Avoidance Decision of Maritime Autonomous Surface Ships Based on Deep Reinforcement Learning” as the selected topic to investigate the collision avoidance problem between MASS and manned ships based on the key technologies in ship intelligent collision avoidance.Using Support Vector Regression(SVR),deep reinforcement learning and other cutting-edge algorithms,the problem of offline and online prediction of manned ship trackings,collision risk assessment and collision avoidance decision making between MASS and manned ships are analysed and studied.The manned ship tracking prediction model enables MASS to gain insight into the behavioural intentions of manned ships(target ships),aiding collision avoidance decision making and helping collision avoidance decisions more forward-looking.The collision risk assessment model can estimate the collision risk with manned ships and support the collision avoidance decision making.The main research works in this thesis are shown below:(1)An Adaptive Chaos Differential Evolution(ACDE)–SVR based offline prediction model for ship trackings is proposed based on the characteristics of SVR applicable to small samples and non-linearity.The ACDE–SVR,recurrent neural network and back propagation neural network are used to build the tracking prediction model and compare the prediction accuracy.The results show that the ACDE–SVR–based ship tracking prediction model has higher and more stable prediction accuracy,it requires less time,and is simple,feasible and efficient.To address the shortcomings of this offline tracking prediction model,an online ship tracking prediction model based on selection mechanism and Least Squares Support Vector Regression(LSSVR)with multiple outputs is proposed.The single output of the traditional LSSVR algorithm is improved to multiple outputs to address the difficulty of applying the traditional single output of the LSSVR algorithm to complex multi–output models.To solve the problems that the LSSVR model is not sparse and the high computational complexity of offline model real-time prediction,an online LSSVR model is proposed and the matrix trick is used to transform the complex matrix inverse operation into an iterative solution,which improves the computational efficiency.In addition,an online model selection mechanism is introduced.When the prediction error of the offline model is lower than the error threshold,the online model is not enabled,and if it is higher than the error threshold,the online model is used.When new samples arrive,in order to reduce the impact of large sample growth on the computational complexity,an elimination mechanism is introduced,and a pruning algorithm is used to remove the initial samples that have a small impact on the model.The comparison experiments demonstrate that the online model can guarantee high prediction accuracy and improve prediction efficiency in the case of small samples,which is a proven method for online prediction of trajectories.(2)A novel model for assessing the risk of ship collisions is proposed in this study.To develop the model,a fuzzy quaternion ship domain model is established for both the own ship and the target ship,and two key parameters are introduced: the maximum interval between the two ship domains and the degree of violation in the two ship domains.These two parameters are solved using support vector classification and geometric methods,and a ship collision probability model is established based on them.Moreover,the consequences of ship collisions are calculated using a computational model that takes into account the relative mass,ship type,speed,and location of collision damage between two ships,based on the law of conservation of momentum.Finally,a ship collision risk assessment model is developed that integrates both the collision probability and collision consequences.And a comparative experiment was conducted with the ship collision risk assessment method based on the ship field,the distance to closest point of approach / time to the closest point of approach parameters,and the spatial collision risk model.The simulation results proved the effectiveness and superiority of the collision risk assessment model proposed in this thesis,which is an effective ship collision risk assessment method.(3)Based on the advantages of model–based and model–free methods in reinforcement learning,a decision–making model for ship collision avoidance between marine autonomous surface ship(MASS)and manned ship is proposed.The model utilizes S–57 chart information,Dyna framework and Deep Q–Network(DQN).When the navigation task of MASS is determined,a static navigation environment is constructed based on S–57 chart information and rasterized to calculate the navigation safety weight for each grid.The Voronoi diagram and an improved A* algorithm are used to obtain the planned route.Considering the small main dimension of MASS and its vulnerability to wind and current factors,a motion model of MASS is established based on the MMG model which takes into account the effects of wind and current factors.Moreover,AIS data is used to extract the data of the target ship(manned ship)and predict the future trajectory of the target ship to make the generation of collision avoidance decisions more forward–looking.Based on the characteristics of the actual navigation of ships at sea,the state space,the action space,and the reward function of the reinforcement learning algorithm are designed.Then,a decision–making model for collision avoidance between MASS and manned ships based on the Dyna–DQN algorithm is established,and the model is simulated and trained using the constructed static navigation environment.Repeated simulation experiments have been conducted to ensure that the MASS can complete the navigation task with reference to the planned route without colliding with static and dynamic obstacles.At the same time,compared with the decision–making model of collision avoidance based on DQN algorithm,the proposed method can significantly improve the training effect and efficiency.Therefore,the proposed method can be utilized in the intelligent collision avoidance module of MASS and is considered as an effective approach for autonomous navigation and collision avoidance of MASS in complex waters.In summary,this thesis effectively improves the autonomous navigation level of MASS,reduces the collision risk during navigation,and has practical significance for ensuring the safety of MASS in maritime navigation.It provides an important theoretical basis and research support for the application of cutting-edge technologies such as machine learning and artificial intelligence in the field of maritime safety assurance.