Reinforcement Learning Based Multi-path Transmission Control Protocol Optimization

Along with the rapid development of computer network and the communicative technology,communication devices are often multi-homed.For example,smart phones and laptops are typically equipped with multiple network interfaces including WiFi and LTE.Traditional single-path TCP use one interface at a time for data transmission,which cannot make full use of the capacity of multiple interfaces and suffers from se_vere performance problems resulting from single-path transmission.The Multi-path Transmission Control Protocol,(MPTCP)has been standardized by the Internet En-gineering Task Force as an extension of single-path TCP,which enables multi-homed devices to establish multiple subflows for simultaneous data transmission.Compared to single-path TCP,MPTCP can effectively leverage multi-path scenarios,which is more robust.However,in multi-path scenarios,especially heterogeneous network scenarios,paths are diverse in bandwidth,round-trip time and other characteristics,which makes MPTCP suffer from a number of performance problems such as bufferbloat,suboptimal bandwidth usage,head-of-line blocking,throughput degradation,longer application de-lay,etc.Congestion control and packet scheduling are fundamental mechanisms and core contents of MPTCP.Thus,solving the performance issues of MPTCP starts with ad-dressing the two problems:(1)congestion control problem,which is how to adjust the congestion window of each subflow.The congestion window has a direct impact on its subflow's throughput,which also significantly influence aggregate throughput and other QoS metrics.(2)packet scheduling problem,which is how to schedule packets over multiple subflows.MPTCP packet scheduling algorithm determines how the data is distributed onto the subflows.A well-designed packet scheduling algorithm should adapt to the changing path characteristics,optimize traffic distribution over multiple subflows and thus improve the performance of MPTCP.In allusion to these problems,we analyze the drawbacks of existing heuristic con-gestion control algorithms and packet scheduling algorithms:they use fixed control rules based on simplified or inaccurate models of the network.As a result,existing algorithms inevitably fail to achieve optimal congestion control and packet schedul-ing across a broad set of changing network conditions.In order to solve the problem completely,we propose and implement a reinforcement learning based MPTCP conges-tion control algorithm called SmartCC and a deep reinforcement learning based packet scheduling algorithm called ReLeS.SmartCC adopts reinforcement learning techniques to learn optimal congestion control rules under different network conditions.Extensive experiments show that SmartCC outperforms the existing MPTCP congestion control algorithms in terms of throughput,jitter and other QoS metrics.ReLeS represents its scheduling policy by a deep neural network,and trains this neural network based on observations collected in different environments.As experiments show,Compared to existing MPTCP packet scheduling algorithms,ReLeS is more adaptive to the varying of network conditions and traffic patterns,and balance a variety of QoS goals such as maximizing average throughput,minimizing application delay,head-of-line blocking and out-of-order buffer size.