Detecting anomalous Internet clients via behavior profiles and reputations

Traditional network defenses focus on how to detect malicious traffic and separate it from the legitimate one. Due to the lack of information about traffic senders, these defense techniques are significantly challenged in predicting the maliciousness of traffic without taking the risk of processing some traffic first. We believe that identifying anomalous traffic senders and predicting identities of possible future attackers is a more promising approach for early detection and prevention of network attacks.;This thesis presents a comprehensive solution to identify anomalous Internet clients based on client behaviors and reputations. We first profile client normal behaviors from Internet traffic traces and characterize each client host by its normal behaviors in the past. By comparing the newly observed behaviors with the previously profiled ones, we can identify any suspicious changes, which indicate possible compromise or misuse of the host. We propose two anomaly detection techniques using either aggregate or individual profiles of client hosts. Both techniques are evaluated with realistic traffic traces containing past Internet-wide anomalies and our results show that they detect the anomalies successfully with low false positive and small latency. While the client behavior profiles are built from the centralized observations at network core (e.g. at routers), we further design a collaborative framework to track client reputations by aggregating distributed information (complaints) from Internet end servers. Our client reputation system collects attack reports about malicious clients from servers, validates these reports with a reliable mechanism, and aggregates the validated reports to calculate each reported client's reputation. With the hypothesis (supported by other published research) that clients that have behaved maliciously in the past are more likely to misbehave and thus warrant a low trust in the future, we track the variation of each Internet client's reputation to predict its future trustworthiness. The client reputation information is useful for Internet servers to evaluate the risk of interacting with this client. We evaluate the performance of the client reputation system with realistic simulations of various Internet-wide security events, including DDoS attacks and worm spread. The simulation results prove the system can effectively and quickly detect various attack events and accurately identify malicious clients.