Research On MAUVS Capture Strategy Based On Hierarchical Reinforcement Learning

Autonomous Underwater Vehicles(AUV)can be used to accomplish complex and heavy underwater tasks,which makes it an important tool for future marine exploration and underwater energy development.With the development of computer and communication technology,more and more attentions are paid to the Multiple Autonomous Underwater Vehicles System(MAUVS).Compared with AUV,MAUVS has the characteristics of flexibility,efficiency and fault tolerance.Multi-agent collaborative capture is often used as a test platform to test the learning performance of robots.Therefore,MAUVS capture is the hotspot of current research.In this paper,aiming at the problem of MAUVS capture,a new capture strategy is designed.The details are as follows:Firstly,the research background and significance of this subject are introduced in detail,and the status of MAUVS research is summarized.Combined with the research on the status quo of multi-robot capture,the existing problems in the process of capture tasks and the limitations of the capture strategies are analyzed.Secondly,in order to overcome the anti-intelligent capture of target AUV,an MAUVS task allocation scheme for dynamic capture is designed.Meanwhile,a task assignment method based on role and energy constraint is proposed,and then the necessity of this scheme in the dynamic capture task is proposed.Thirdly,the flexible MAXQ algorithm is used in hierarchical reinforcement learning to design a specific capture strategy.Aiming at the defect that MAXQ cannot autonomously layer itself,this paper proposes a method to modify the MAXQ abstraction mechanism by using the Self-organizing Feature Mapping(SOM)neural network,that is,the autonomous layering strategy S-MAXQ.This method uses the characteristics of SOM self-organizing feature mapping neural network so that sub-tasks can be autonomously discovered by the Agent and carry out parallel learning accordingly,which can adapt to the learning tasks in dynamic environment more.The V-REP simulation software is used to build a simulation environment with obstacles to simulate the hierarchical effect of the algorithm,and the performance of the algorithm is analyzed in Matlab.Then,in order to adapt to the complicated marine environment,semi-Markov game theory method is used to realize the hierarchical reinforcement learning of MAUVS.Based on the above-mentioned S-MAXQ,MAUVS behavior prediction function is added and SP-MAXQ algorithm is proposed correspondingly.Meanwhile,a state-action table is established to make the learning experience better reuse,which enhances the ability of MAUVS to adapt to the environment and cooperate with each other in complex environments.The V-REP simulation software builds an obstacle scene to simulate the real environment for capture simulation and the algorithm performance is analyzed in Matlab.Finally,the Pioneer3-DX robot is used to make the entity simulation results,further illustrating the effectiveness of the proposed algorithm.