Defenses against network scanning and other malicious remote host activity: Empirical studies, analysis, and new approaches

Network resources linked to the Internet are susceptible to a variety of attacks that become increasingly hard to detect with the increasing complexity in Internet traffic dynamics and heterogeneity. Even with the assumption that end-to-end Internet dynamics can be correctly characterized, it is exceptionally difficult to identify malicious network traffic, as it may be crafted to adhere to network protocol specifications both syntactically and semantically and to mimic legitimate traffic behaviour. For example, reconnaissance activities may be so crafted, with the objective of gathering information to launch subsequent attacks.;To address the limitations found, we introduce two novel network scan detection algorithms and a new password guessing resistant protocol. Our empirical evaluation argues that they offer practical defenses against such malicious network activity. Our detection and defense mechanisms are designed to capture large-scale events and those launched by adversaries with access to a large number of machines (e.g., a botnet).;As part of our empirical evaluation, to our knowledge, we are the first to explore in detail the problems that can arise when evaluation is based on a ground truth reference rather than absolute ground truth. We model the problem of evaluating detection algorithms in the absence of absolute ground truth, and analyze the requirements of using a ground truth reference for either evaluating one intrusion detector or comparing multiple detectors.;In this thesis, we focus on network scanning and automated password guessing attacks, two types of widespread malicious network activity that are known precursors to a broad range of compromises of machines and accounts. Recently, these activities are often conducted in a large-scale capacity targeting apparently random networks, rather than being directed or strategic. We conduct an analytical and empirical study of these two malicious activities using recent real-world network traces and logs collected at various sites. We examine and evaluate selected detection and prevention approaches to identify their limitations and strengths. For network scan detection, our results show that there is often a crucial trade-off between detection and false positive rates, due to a lack of both a built-in algorithmic adaptability and a manual parameterization criterion based on the deployment environment. For password guessing attacks against existing login protocols, we find there is a fundamental trade-off between user login convenience and login security with respect to password guessing.