Adaptive Network Congestion Control Methods Based On Reinforcement Learning

The rapid development of next-generation Internet technologies and the rapid increase of Internet applications and services(live video,online games,AR/VR,etc.)have brought convenience to people,improved quality of experience,and put forward new requirements on network performance.Network congestion control is an essential tool to improve network throughput,which can lower data transmission delay and packet loss rate,reduce network congestion,enable effective utilization of network resources,and provide users with a high quality of service experience.However,the performance of the currently used TCP congestion control protocol is deteriorating,which will seriously affect the development of the network and the quality of user experience due to the dramatic increase in the number of network users.Therefore,it is of great practical importance to apply machine learning technology to congestion control to improve the network's performance by adaptively controlling network congestion according to the current state of the network.This thesis presents a study of congestion control protocols for improving network performance in computer networks.First,traditional congestion control protocols use predefined rules to perform congestion control based on network signals,which cannot adapt to various network scenarios.They cannot use the experience to adjust congestion control adaptively based on network state changes,thus resulting in low network performance.The thesis applies deep reinforcement learning algorithms to congestion control and proposes an adaptive congestion control protocol based on deep reinforcement learning.Second,the reinforcement learning-based congestion control protocols can be adaptively adjusted according to the network state,but they require many iterations to measure the congestion signals,which results in slow model convergence and under-utilization of bandwidth.The thesis presents a deep reinforcement learning-based staged high-throughput congestion control protocol to improve reinforcement learningbased protocols' training efficiency and convergence speed.Finally,since the reward function of reinforcement learning is fixed,it is not easy to make the reinforcement learning-based congestion control protocol meet the performance requirements of different types of applications without redesigning the reward function and training new models.In order to meet the performance needs of different types of applications,the thesis designs a multi-objective congestion control protocol based on reinforcement learning.The detailed research work and contributions of this thesis are summarized as follows:(1)A reinforcement learning-based congestion control protocol to adapt to multi-typed networksTraditional congestion control protocols are designed for one specific type of network and cannot achieve good performance in other types of networks.For example,the TCP NewReno algorithm,designed for wired networks,regards the loss of transmission data as a significant signal of network congestion.However,the loss of transmitted data in a wireless network can be caused by errors occurring on the link.Therefore,once packet loss is all reduced in the congestion window in wireless networks,that will result in under-utilization of link bandwidth and lowered network performance.The thesis designs a congestion control protocol adapted to multi-typed networks,called Glider,based on deep reinforcement learning by leveraging the network performance metrics.Glider can take appropriate actions to adjust the size of the congestion window according to the changes in the network environment(including network throughput,latency,and packet loss rate)to achieve optimal network performance.This enables the congestion control protocol to achieve the application-level goals of high throughput,low latency,and packet loss.In order to apply discrete-based deep reinforcement learning to a continuous network environment,the thesis designs a dynamic bisection division algorithm to discretize the packet transmission process into steps for ensuring the feasibility of Glider on congestion control.We have used an extensive array of experiments on AWS cloud services to show that Glider can achieve 1.4× higher throughput than BBR,while its average latency is 43 ms,much lower than congestion control protocols such as Fillp Sheep,BBR,Copa,PCC,PCC Vivace and Indigo.(2)A staged high throughput congestion control protocol based on reinforcement learningExisting congestion control protocols based on reinforcement learning,on the one hand,need to improve the network throughput and convergence speed according to the network objectives.On the other hand,many iterations are required to measure the congestion signals,resulting in slow convergence,under-utilization of resources,and poor network performance.The thesis proposes a reinforcement learning-based staged high-throughput congestion control(HTCC)protocol that utilizes a fast-growth algorithm and a PPO reinforcement learning-based congestion control method to address these problems.HTCC can speed up the network's convergence and make full use of reinforcement learning to adapt the congestion window.As a result,it improves the convergence speed and throughput of the model under different network conditions while also lowering data transmission and packet loss.A comprehensive set of experiments based on the network opensource simulation platform Pantheon in a 100 Mbps network show that the HTCC congestion control protocol has a throughput of 91.69 Mbps,which is relatively close to that of the TCP Cubic and Copa protocols,but exceeds that of TaoVA-100x by 11.6%,PCC by 58.2%,PCC Vivace 73%,and the latency of HTCC is 55.37 ms lower than TCP Cubic by 79%and lower than Copa by 70%.(3)A multi-objective congestion control protocol based on reinforcement learningExisting reinforcement learning-based congestion control protocols may fail to meet the performance requirements of different types of applications simultaneously.Specifically,since the reward function of reinforcement learning is fixed,reinforcement learning algorithms need to redesign the reward function and train new models to satisfy different types of applications.The thesis develops a novel multi-objective congestion control protocol based on reinforcement learning,called MOCC.Firstly,to address the problem of slow convergence of the model caused by using a fixed initialization congestion window,MOCC proposes a bandwidthbased dynamic initialization congestion window size method to enable MOCC to converge to a stable state quickly.Secondly,to address pseudo-interruption at the end of the episode in reinforcement learning,MOCC designs a method to interrupt reinforcement learning episodes based on network state to improve the training efficiency of multi-objective reinforcement learning by appropriately interrupting the episode.Finally,the trained MOCC congestion control policy can simultaneously meet the performance requirements of different applications by setting preferences for multi-objective reinforcement learning without redesigning the reward function and training model.MOCC trains a single policy network capable of optimizing the entire preference space for congestion control,allowing the trained model to produce the optimal policy for any given preference.Thus,MOCC fundamentally changes the design of existing protocols where the reward function is fixed and offers a more significant advantage in catering to different types of applications.Experimental results based on a Linux kernel simulation tool in an unseen network environment with varying bandwidths show that the MOCC congestion control protocol has much higher link utilization than Vegas and BBR,while its latency is 26.02ms,11.22%lower than BBR,72.64%lower than Vegas.