On Congestion Control For Computer Networks Based On Reinforcement Learning Theory

With the rapid development of Internet and the sudden increase of computer network users, various network applications appear almost every day. And the network congestion led by the sharp increase of net flux has been a bottleneck problem, which restricts the development and application of network. The congestion of information is the main reason that affects the quality of service (QoS) in network. Therefore, it is important to solve the congestion problem effectively to improve the performance of network. But network is a complex large system with respect to the nature of time-varying and uncertainty, and the complexity and accuracy of mathematical model can not match the real time requirement of network usually. So a kind of congestion control algorithm based on learning mechanism is needed in order to obtain better congestion control effect.The reinforcement learning algorithm is independent of the mathematic model and priori-knowledge of controlled object. It obtains the knowledge through trial-and-error and interaction with the environment to improve its behavior policy. So it has the ability of self-learning. The reinforcement learning is very suitable for complex time-varying network system. So in this dissertation, some congestion control algorithms are proposed based on reinforcement learning theory in order to solve the problem of congestion control for networks. The main innovative contributions of this dissertation are summarized as follows.Based on the adaptive heuristic critic algorithm in reinforcement learning theory, a hierarchical reinforcement learning ABR (available bit rate) flow controller is presented for the problem of congestion control in ATM (asynchronous transfer mode) networks with single bottleneck node. The action selection element of the controller considers the queue size in buffer and cell lose ratio respectively based on hierarchical mechanism. The sending rate of ABR is obtained as a weighted combination of the decisions generated by the two sub-elements. Then, the learning process of the parameters in the controller is designed based on the simulated annealing algorithm to accelerate the learning speed, and the problem of the local extremum is avoided effectively.Based on the idea of Q-learning in reinforcement learning theory, an ABR controller is presented for the problem of congestion control in ATM networks with two bottleneck nodes. The controller transforms the problem of searching for the optimal control policy to the problem of searching for the optimal matrix H through the design of Q-function without the parameters of network model. The matrix H is learned based on RLS (recursive least squares) and the control policy is obtained with the optimal performance index.Based on the Q-learning algorithm in reinforcement learning theory, an active queue management (AQM) algorithm is presented for the problem of congestion control in TCP (transmission control protocol) networks. The controller learns the Q-values corresponding to each state-action pair in TCP network, and adjusts the learning rate by the confidence value which is a measure of how closely the corresponding Q-value represents the current state of network. Then, the state space of network is predigested by the transformation of state space. The action selection policy is improved by Metropolis criterion to cope with the balance between the exploration of unknown space and exploitation of the knowledge having been achieved. Secondly, the proposed controller is applied to the network with multiple bottlenecks network based on the cooperative reward.Based on the fuzzy Q-learning algorithm, an AQM algorithm is presented for the continuous state space in TCP network. In the learning process, both the actions selected by the learning agent and Q-values are inferred from fuzzy inference system. Then, the consequent parts of fuzzy rules are optimized by the genetic algorithm, and the optimal action for each fuzzy rule is obtained through the optimization process.Based on the Nash Q-learning algorithm, a flow controller is presented for the networks with non-cooperative users. Different price levels are determined for different services and different QoS requirements in the same service based on the pricing scheme and are used in the calculation of reward. The learning process is executed through the selection of Q-values satisfying the Nash equilibrium condition. The uses select the sending rate as high as possible when the performance of the entire network is optimal.For the problem of routing selection in network, firstly, a dual metrics Q-Routing algorithm is presented. The algorithm takes the transmission time of packets and the cost of link as Q-values, and learned them respectively. The routing selection decision is determined by the weights considered the two metrics. Secondly, a memory based Q-learning routing algorithm is presented. The Q-values corresponding to the link reflect the states of network through the learning process. The learning agent predicts the traffic trend through the best Q-values learned kept in the memory and the recovery rate for the link have congested before, and determines the selection of routing strategy.