Research On Intelligent Multicast Routing Based On Reinforcement Learning In Software Defined Networks

With the rapid development of network infrastructure,there are more and more scenarios in network applications where data needs to be transmitted to multiple recipients simultaneously.If traditional unicast transmission is used in these scenarios,the source node needs to send multiple copies of data to each destination node,which not only wastes a large amount of network resources but also easily causes network congestion.In contrast,using multicast technology can better solve these problems because multicast data only needs to be transmitted once along the same path,rather than multiple times,resulting in more efficient data transmission.To determine multicast transmission routes,an optimal multicast tree needs to be constructed,along which multicast data will be transmitted.Efficient and reasonable design and implementation of multicast routing rely on timely and convenient network state information collection and acquisition,as well as the design of multicast tree construction algorithms.Compared to traditional network measurement methods for obtaining network state information,Software-Defined Networking(SDN)technology can effectively address the problems of limited network state information,poor adaptability to dynamic and complex network changes,and inflexible data forwarding through highly flexible and programmable global controllers.Currently,the algorithms for constructing multicast trees mainly include classical algorithms,intelligent optimization algorithms,and reinforcement learning methods.However,classical and intelligent optimization algorithms usually lack consideration of global dynamic network changes when constructing multicast trees and face problems such as high time cost and difficulty in converging to the global optimum.Reinforcement learning methods achieve great performance improvement in constructing routing paths by learning to maximize reward values through the interaction of agents and environments.However,current reinforcement learning methods used in multicast research only consider the simple case of hop count information to ensure convergence,failing to fully and comprehensively utilize multiple network factors.To address these issues,this paper proposes the use of deep reinforcement learning and deep hierarchical reinforcement learning in SDN to construct multicast trees.The main work and innovations of this paper are as follows:(1)Proposed a deep reinforcement learning intelligent multicast routing algorithm based on deep Q network for constructing multicast trees in SDN.To solve the problem that the original method cannot fully utilize network status information,SDN technology is used to flexibly collect global network status information,and multicast tree state matrix,link bandwidth matrix,link delay matrix,and link packet loss rate matrix are designed as the state space of the reinforcement learning agent.Secondly,all links in the network are designed as the action space of the agent,and four action selection strategies are designed for four situations.Single-step and final reward functions are designed to guide agent decisions,and the optimal multicast tree with the best indicator is finally constructed.To improve the learning efficiency and convergence effect of the agent,double network structure,dueling network structure,and experience prioritized replay strategy are adopted.Finally,the SDN controller traverses the multicast tree topology information constructed by the agent in reverse and sends it to the SDN switch in the form of flow tables to implement intelligent multicast routing.Experimental results show that the proposed algorithm can make more intelligent multicast routing decisions in dynamic changing network environments,and the constructed multicast tree has better bandwidth,delay and loss rate performance.(2)An SDN intelligent multicast routing algorithm based on deep hierarchical reinforcement learning is proposed to address the problems of redundant branches,excessively large action space,and slow convergence in the algorithm designed for problem(1)during the construction of multicast trees.The algorithm decomposes the problem of constructing a multicast tree into two sub-problems: the upper-level sub-problem is to select the branching node,and the lower-level sub-problem is to construct the optimal path from the branching node to the destination node.By utilizing the characteristics of SDN’s global network awareness,multicast tree status matrix,link bandwidth matrix,link delay matrix,link packet loss rate matrix,and sub-target matrix are designed as the state space of the inner and outer controllers.Different action spaces are designed for the upper and lower levels,where the meta-controller considers all nodes in the network as the action space,and the inner controller considers the neighboring edges of the current node as the action space.Four action selection strategies are designed to construct paths.Furthermore,five reward strategies are designed for the inner and outer controllers to guide the intelligent agent to construct the optimal multicast tree.The experimental results show that compared with the algorithm in(1),the efficiency of searching for redundant branches is higher,and the constructed multicast tree has better performance in terms of bandwidth,delay,and loss rate.