Research On Adaptive QUIC Congestion Control Algorithm Based On Reinforcement Learning

Currently,computer networks remain the primary medium for information transmission and communication,where the Transmission Control Protocol(TCP)plays an important role.The TCP protocol has been extensively studied by many researchers and the network performance has improved accordingly.However,with the development of wireless technology and the emergence of modern applications,shortcomings of TCP such as queue blocking and protocol rigidity have gradually emerged.Recently,Quick User Datagram Protocol Internet Connection(QUIC),a modern transport layer protocol,has attracted a lot of attention.QUIC implements TCP-like features based on User Datagram Protocol(UDP)in user mode and can effectively address the poor performance of TCP in networks with high packet loss rates.However,QUIC still uses traditional congestion control algorithms in TCP,which makes it impossible to fully exploit QUIC’s advantages.Traditional congestion control algorithms are mainly designed for specific network scenarios,suffer from poor performance when separated from the target scenario,fail to fully utilize the network bandwidth,and are sensitive to packet loss,among others.Therefore,there is strong research value in how to improve the congestion control algorithm of the QUIC protocol and improve its performance in different network environments.Thus,the main work of this paper is as follows:(1)Aiming at the robustness of congestion control algorithm in multi-stream scenarios of QUIC protocol,a congestion control algorithm based on the combination of deep reinforcement learning algorithm and traditional congestion control algorithm Bottleneck Bandwidth and Round-trip propagation time(BBR)is proposed,so as to improve the performance of QUIC protocol in different network environments.First,a network link model is constructed to maximize the overall throughput and minimize the packet loss rate under delay constraints.Since most of the environmental states are continuous variables,this problem is modeled as a Markov decision process.The reward function is designed under the guidance of the optimization objective,and the neural network is trained with historical data to learn a mapping between network state and transmission performance.Finally,simulation experiments are conducted to demonstrate the effectiveness and superiority of the proposed algorithm.It is shown that the proposed algorithm can improve the throughput performance and reduce the link delay of the QUIC protocol under various network conditions compared to the existing congestion control algorithms under the QUIC protocol.(2)Aiming at the problem of fair bandwidth allocation in QUIC multi-stream scenarios,a fair congestion control algorithm based on multi-agent deep reinforcement learning is proposed.First,we introduce the Jain fairness index to measure the fairness of the bandwidth allocation and identify the optimization objective of maximizing the singlestream performance with the Jain fairness index.The game relations between multiple QUIC flows are analyzed in,where the problem is modeled as a competition-cooperation problem.The streaming states are combined into joint states and the multi-stream competitive bandwidth scenario of the QUIC protocol is modeled as a Markov game.A congestion control strategy is designed based on a combination of a deep multi-agent reinforcement learning algorithm and a congestion control mechanism.From the optimization objective,a reward function is designed,which includes a local reward to represent the single-stream performance and a global reward to represent the fairness of the bandwidth allocation.Considering that there is no information interaction in the multistream environment,a centralized training and distributed execution model architecture is adopted.Simulation results show that the proposed algorithm can improve the fairness of bandwidth allocation and guarantee the overall bandwidth occupancy compared to the existing QUIC congestion control algorithm.