Enhancing security in an IP backbone network

As the Internet continues to grow in size and complexity, the challenge of effectively provisioning, managing, and securing it has become inextricably linked to a deep understanding of Internet traffic. Due to vast amount of data, and wide diversity of end-hosts and services found in Internet traffic, we need techniques that can extract underlying structures and significant communication patterns.; In this dissertation, we propose a systematic methodology for profiling Internet backbone traffic that (1) not only automatically discovers significant behaviors (communication patterns) of interest from massive traffic data, (2) but also provides a plausible interpretation of these behaviors to aid security analysts in understanding and quickly identifying anomalous events of significance. For these purposes, a combination of data mining and information-theoretic techniques are employed to automatically cull useful information from largely unstructured data. An entropy-based adaptive algorithm is developed to extract significant clusters of interest. We introduce a behavior classification scheme that automatically groups clusters into classes based on communication patterns and feature distributions using relative uncertainty. In addition, we use dominant state analysis to uncover cluster structure for interpretive analyses. The analysis of traffic data collected from a variety of links at a large IP backbone network shows that the approach indeed provides a robust and meaningful way of characterizing and interpreting network behavior.; Given unwanted traffic revealed in exploit behavior profiles, we develop simple yet effective blocking strategies an IP network may pursue to reduce substantial exploit traffic. To demonstrate the operational feasibility, a real-time traffic profiling system has been designed and implemented. Experiment results show that under normal traffic conditions, resources on a commodity PC are sufficient to continuously process flow records and build behavior profiles for high-speed links in operational networks. For sudden traffic surges caused by events such as denial of service attacks or worm outbreaks, a novel profiling aware filtering algorithm is proposed to reduce the CPU and memory cost of the real-time system while maintaining high profiling accuracy. Thus, the profiling system can become an effective tool for security analysts with applications to critical problems such as detecting unknown security exploits and profiling unwanted traffic.