Research On QoS Adaptive Routing Algorithm Based On Deep Reinforcement Learning

With the diversified development of application scenarios in the future network,the business type has changed from a single data transmission mode to multiple transmission modes such as video,data,interactive and current affairs multimedia.The requirements of user present multi-dimensional and differentiated characteristics.Different business types have different requirements in terms of delay,packet loss rate,cost overhead,throughput,security,reliability and so on.The core of realizing QoS guarantee for service flow is the precise matching of network resource and service index requirement.As an important part of the network architecture,routing plays an extremely important role in meeting the requirements of user.The problem to be solved in this paper is to enable routing nodes in diversified network business scenarios to have differentiated service capability,to generate on-demand routing strategies based on the QoS indicators of service flow,and to allocate network resource reasonably.This paper combines deep reinforcement learning with network routing,and designs ORL,an on-demand routing learning model with a three-layer logical plane structure.This model can generate on-demand routing strategies for service flows with different QoS index requirements.A QoS adaptive on-demand routing algorithm Od RDDPG is designed and implemented in ORL model.This algorithm is better than traditional routing algorithm,but there is still room for improvement.This paper uses two schemes to optimize the algorithm.The first scheme is to combine Od R-DDPG algorithm with traffic engineering,and design Od R-DDPG-LSE,a QoS adaptive on-demand routing algorithm based on link congestion state estimation.First,estimate the link congestion state according to end-to-end performance of the network,then generate action noise according to the congestion state estimation,so as to guide the exploration direction of action space and improve the learning efficiency.The second scheme is to select a better network model.We design two on-demand routing algorithms Od R-TD3 and Od R-SAC based on TD3 and SAC.Od R-TD3 adopts dual-Q network to solve the problem of overestimation of action value.Od R-SAC algorithm uses the principle of maximum entropy to expand the exploration scope of action space and prevent prematurely falling into the local optimal value.Finally,this paper conducts three groups of experiments in the network simulator Omnet++ and Keras framework,and uses different data sets to evaluate the performance of Od R-DDPG,Od R-DDPG-LSE,Od R-TD3 and Od R-SAC algorithms.The experimental results of the first two groups show that these algorithms are better than DV algorithm and SPF algorithm in terms of delay,jitter,packet loss rate and qualified rate.These algorithms achieve better robust convergence and can provide on-demand routing service according to the QoS indicators of the service type.Among them,Od R-DDPG-LSE algorithm has the best performance,which achieves the best routing strategy with fewer steps.Compared with other algorithms,the average delay of Od R-DDPG-LSE is reduced by at least 10.5%,the average jitter is reduced by at least 3.5%,and the average packet loss rate is reduced by at least 15.2%.The third group of experiments comprehensively evaluate the performance of these on-demand routing algorithms under different traffic intensity.The experimental results show that these algorithms are more suitable for solving the situation of relatively high traffic intensity or even some congestion.The algorithm with the best performance is still Od R-DDPG-LSE algorithm.