| Network fault and security management are essentially control problems in the network system. Control actions should be taken by widely distributed network management agents. In this dissertation, we develop a generic framework for such agents to learn to perform proactive network management. The decision engine of the intelligent agent is formulated as a Partially Observable Markov Decision Process (POMDP), which expressively models uncertainties in both state evolution and observations. Although POMDP is more generic and effective than other simpler modeling frameworks, it poses some technical problems in practice. Solving POMDP is computationally demanding and only problems of small size can be solved exactly. Firstly, we present a novel learning approach, in the context of Reinforcement Learning (RL), to obtain sub-optimal policies for POMDP problems of realistic size, even without explicit models. Secondly, we identify the partial observability and cost functions that exist in typical network fault and security management and propose to optimize management actions by using POMDP policies. More specifically, in order to scale up a POMDP diagnostic controller for large scale network, we represent the observation function as a Bayesian network and learn the structure and parameters of Bayesian network from the historical dataset; in order to prevent fault propagation and stabilize the restoration process in a link-state routed network, we use POMDP policy to throttle the flooding of link state packets; We also study the control policy to detect and prevent denial-of-service (DoS) attack. Some relationships between the signature of DDoS attack and the traffic aggregate recorded by MIB variables are identified. The detection and pushback agents can also be driven by POMDP policy. Finally, we validate our results by extensive simulations. |
