Deep Reinforcement Learning Based MAC Protocols For Heterogeneous Underwater Sensor Networks

In recent years,due to the accelerated exploration of marine resources,underwater communication has been widely applied in science,the military,and commerce.As a powerful tool to explore underwater resources,underwater sensor networks(UWSN)have attracted great attention and have become the focus of research in the field.As one of the key technologies for constructing underwater sensor networks,the media access control(MAC)protocol is not only a necessary condition for controlling and managing all nodes in the network to share communication channels,but also the focus of attention in the field.First,this thesis briefly investigates the network topology,communication characteristics,and the current development of UWSN.On this basis,the theoretical basics of deep reinforcement learning(DRL)algorithm and the current researches on underwater MAC protocols are introduced.Meanwhile,we emphasized the main ideas of some typical underwater MAC protocols.Secondly,a deep reinforcement learning multiple access for heterogeneous underwater multi-channel acoustic sensor networks(UM-DLMA)is proposed.In this protocol,nodes using the UM-DLMA protocol are referred to as agents,while others are called non-agent nodes.Nonagent nodes can send data packets to a relay node through the multiple acoustic channels they have been assigned,while agents haven’t been assigned to specific channels for data transmissions.Through sufficient training and learning,agent nodes are usually capable of capturing the underutilized channels of non-agent nodes,and they are inclined to select the underutilized channels,which are beneficial to maximize the network performance for additional data transmissions.When designing the reward function,the compensation capability value is introduced to characterize the priority for the agents to select different channels for data transmission.Meanwhile,agents are encouraged to transmit data packets on the channel with a large compensation capability value.This protocol can fully utilize the free channels to maximize the total network throughput and channel utilization.Subsequently,an optical and acoustic multi-channel deep reinforcement learning multiple access for heterogeneous underwater acoustic-optical hybrid sensor networks(OA-DLMA)is proposed.The nodes using the OA-DLMA protocol are referred to as agents,and the other nodes are named non-agent nodes.Non-agent nodes send packets to the relay node through the pre-assigned acoustic or optical channels,while the agent nodes are not assigned to specific channels.The agents can learn the transmission mechanisms of non-agent nodes.In this way,agents can capture free acoustic or optical channels and send data packets on them.Moreover,since the optical channel has a higher capacity and low delay,a differentiated reward mechanism is designed to encourage agents to preferentially choose the underutilized optical channel for data transmissions.Thus,the transmission on an optical channel will have a larger reward value than on an acoustic channel.This protocol enables the agents to capture the available channels and select the most appropriate channels to transmit data packets.As a result,the acoustic and optical channels can be fully utilized to maximize the total network throughput and channel utilization.Finally,the proposed protocols are evaluated by extensive simulation experiments.The simulation results show that the proposed algorithm can maximize the total network throughput and channel utilization,and its performance is much better than that of the related comparison algorithms.