Programmable and Closed-Loop Traffic Measurements for Better Accountability and Security in Future Networks

Accurate traffic measurement and monitoring is keystone in a wide range of network applications such as detection of anomalies and security attacks, and traffic engineering. A number of critical network management decisions such as blocking traffic to a victim destination, re-routing traffic, or isolation of anomalies, require extraction of real-time statistics from network traffic. A high-quality network measurement tool is crucial for extracting such patterns of interest and making informed decisions to ensure proper network operation. Traditionally, the measurements are performed using open-loop mechanisms that base their decision by using a limited sample of the streaming data. The open-loop techniques not only suffer from inaccuracies due to sampling, but the high speeds and complexity of today's networks, coupled with ever evolving threats, render the traditional open-loop paradigm inadequate for real-time and accurate analysis. This work proposes, implements, and evaluates an alternate closed-loop measurement paradigm, that offers higher accuracy and speeds in network measurements and analysis from the traditional mechanisms.;The closed-loop measurement system demands high levels of processing and programmability where streaming network traffic can be processed and new refined measurement requirements be dynamically evaluated and programmed in real-time. The work addresses the requirements and offers novel hardware mapped architectures and software based streaming analysis and control algorithms that are also integrated in a hardware-software co-designed closed-loop measurement system. The hardware based solutions include statically and dynamically configurable parallel and pipelined measurement architectures, that are mapped on a commodity Field Programmable Gate Array (FPGA). An evaluation of the architectures demonstrates high degrees of programmability and processing of streaming network traffic. The work also makes novel use of fine-grained Partial Dynamic Reconfiguration (PDR) in FPGAs in offering an innovative statically-mapped and dynamically-configurable measurement unit, the Dynamically Reconfigurable Socket, that increases the processing capacity by an order of magnitude from static solutions, while offering much reduced reconfiguration latencies over conventional PDR mechanisms.;The work also proposes novel traffic streaming algorithms to support streaming analysis and control of the closed-loop measurements. The algorithms cater to varying degrees of computational and storage budgets, detection latencies, and accuracies in answering users' measurement requirements. A key goal of the algorithms has been to come up with representative set of refined measurement requirements that can best utilize the limited hardware based measurement resources. The work also examines the use of randomized sketch based algorithms in automating the scope of closed-loop measurements, thereby increasing their resiliency against adversaries working to outsmart detection of network anomalies behind the regular framework of non-randomized algorithms. The proposed architectures and algorithms are integrated and evaluated in a hardware-software co-designed system that demonstrates high levels of programmability and accuracy in measurements, while offering low processing latencies in detecting and isolating hard to track local and distributed anomalies, such as heavy-hitters, global icebergs, and port-scans.