Deep Reinforcement Learning Based Multi-object Behavioral Decision-making For Maritime Autonomous Surface Ship

This research focuses on the adaptive navigation of maritime autonomous surface ships(MASS)in an uncertain environment.To achieve intelligent obstacle avoidance of MASS,a driving behavioral decision-making model based on hierarchical deep reinforcement learning is proposed.The proposed model is mainly composed of two layers:the navigation situation awareness layer and behavioral decision-making layer.The navigation situation awareness layer mainly quantifies the sub-scenarios according to the International Regulations for Preventing Collisions at Sea(COLREG).The role of the behavioral decision-making layer is to learn the environmental state in a quantized sub-scenario to train the driving strategy.A driving decision-making algorithm based on deep Q-learning is designed utilizing the environmental model,ship motion space,excitation function,and search strategy to achieve behavioral decision-making and obstacle avoidance for MASS.Finally,two sets of verification experiments of the deep reinforcement learning(DRL)and improved DRL algorithms are designed with Rizhao port as a study case on Python and Pygame.Moreover,the experimental data are analyzed in terms of the convergence trend,iterative path,and collision avoidance effect.The experimental results show that the APF-DRL algorithm can safely avoid obstacles and decide autonomous driving behavior,and further demonstrates the effectiveness and applicability of the algorithm designed in this paper.The innovative results of this research are as follows:(1)Applying ontology,analyzing multi-source heterogeneous information from both entity class and attribute,designing the MASS navigation situation ontology conceptual model to understand the navigation situation,and quantifying the divided scenes in combination with International Regulations for Preventing Collisions at Sea(COLREGS).Process and build a Prolog rule base for collision avoidance decisions.(2)Design a multi-objective DRL-based behavioral decision-making algorithm for MASS.Among them,the discrete decision-making space of MASS behavioral,the reward function consisting of safe obstacle avoidance and close target points,behavioral selection strategy,state value function and other elements are designed to realize the MASS navigation behavioral decision-making under uncertain environment.(3)Combined with the gravitational field concept of APF,the APF-DRL behavioral decision-making algorithm is designed to improve the slow iteration and easy to fall into local iteration of DRL.