Research On Maritime Network Resource Allocation Based On Deep Reinforcement Learning

The maritime information management system is developing rapidly towards automation and intellectualization since the proposing of "smart maritime".However,providing reliable and high-quality network service quality(Quality of Service,QoS)for marine users has become a bottleneck restricting the development of marine communications.In the ocean environment where substantive heterogeneous networks coexist,the traditional routing algorithm only has a single optimization objective for QoS and needs to repeatedly establish model and adjust the parameters,thus unable to meet the demands of most businesses for QoS and online information acquisition.Due to high complexity,the optimization of QoS for multiple objectives(i.e.,delay,packet loss,energy)is an intractable problem to the traditional methods.Therefore,under the constraints of multi-objective QoS,routing optimization of marine wireless network and efficient allocation of resources are delved into in this paper.(1)In view of the heterogeneity in maritime communication networks,a software-defined maritime communication network architecture is proposed.It integrates deployment of space,air,land and sea network,and uses the software-defined network to break the vertical structure of open system interconnection(Open System Interconnection,OSI)model,thereby realizing the separation of control and retransmission,and simplifying the operation,maintenance and management of network.Secondly,its unified Openflow standard realizes concentrated scheduling of data and solves the communication problem in heterogeneous maritime network,thereby raising the efficiency of transmission.(2)For the problem of routing optimization in the proposed SDN architecture,a scheme of optimal link selection based on Markov decision processes(Markov Decision Processes,MDPs)is proposed.It takes the channel condition in the marine communication environment,cache state of node and the energy consumption into comprehensive consideration.Then the system evaluation model is established to evaluate the quality of node and link.In this scheme,the data transmission processes are modeled as MDPs,and the SDN controller gets through agent to independently explore the system environment to learn the transmission information of external environment and obtain the optimal strategy.When the agent finds a certain rule,the system will carry out dynamic routing optimization.Finally,the best scheme is made.(3)In order to achieve rapid allocation and guarantee the QoS of business information among multiple nodes(delay,total throughput,link reliability),a multi-objective QoS optimization mechanism based on Deep Reinforcement Learning(Deep Reinforcement Learning,DRL)is proposed.It uses the DRL algorithm in the routing decision meta-layer of SDN.In the early stage,it obtains the optimal path offline through the optimal link scheme proposed above,and then inputs the corresponding node set into the routing database,and uses the node set as sample label to train the DRL model.After the optimal model is trained,the optimal path is obtained only by simple calculation with the given weight parameters in the interior of model when there is a new connection request,thereby significantly raising the routing efficiency and realizing rapid allocation of network resources.(4)The feasibility and superiority of the proposed scheme and mechanism in the designed complex marine communication environment are verified through simulation experiments.The comparison experiments with the existing approaches shows that the proposed scheme can efficiently make the optimal decisions and the proposed mechanism greatly improves the routing efficiency in data transmission,while performance indicators such as throughput,packet loss rate,and energy consumption are also guaranteed.