Adaptive Cyber Defenses to Mitigate Botnet-Borne Threat

A botnet is a network of compromised machines remotely controlled by an attacker. Over the past two decades, the modus operandi of botnets has evolved to facilitate a wide array of attacks ranging from stealthy exfiltration of sensitive data to large-scale Distributed Denial-of-Service (DDoS) attacks. As a result, botnets have emerged as one of the biggest threats to the Internet ecosystem.;Despite several successful efforts at shutting down botnet operations, attackers quickly reconstruct them while switching to more stealthy and resilient architectures. To address this ceaseless arms race between the defenders and the attackers, a new paradigm known as Adaptive Cyber Defense (ACD) has emerged as a promising approach. ACD techniques mitigate attack campaigns by increasing the cost and complexity for a malicious actor to successfully execute an attack. In this thesis, I present ACD models and techniques to address two significant threats: data exfiltration and DDoS.;Botnets that facilitate data exfiltration persist in the target network for an extended period of time and operate in a stealthy manner. In the recent past, attackers that employ botnets to exfiltrate data have implemented a new form of stealth---known as architectural stealth---in which the attacker constructs a communication architecture that reduces the exposure of malicious traffic to the detectors. To disrupt such botnets in a resource-constrained environment,we first develop two dynamic monitoring strategies that significantly increases the likelihood of intercepting bot traffic. Next, we design a detection mechanism, namely DeBot, that processes the intercepted traffic and exploits known intrinsic behaviors of bots to detect them. Finally, to minimize the persistence of such botnets within a network, we develop a Reinforcement Learning (RL) model that combines the proactive capability of honeypots to prevent lateral movement and the reactive nature of detection mechanisms to detect persistent bots within the network. We provide a proof-of-concept of the proposed techniques, and study their performance in a simulated environment. The results show that the proposed approaches are promising in protecting a network against long-term exfiltration campaigns by stealthy botnets.;On the other end of the attack spectrum, adversaries also employ botnets to launch large-scale volumetric DDoS attacks. To mitigate the impact of DDoS attacks, organizations are increasingly adopting proxy-based architectures. These architectures introduce a well-provisioned intermediate layer of secret proxies between end users and target services and reduce the impact of a DDoS attack by migrating the clients to new proxies and shuffling the clients across proxies so as to isolate malicious clients. However, the reactive nature of these solutions presents a weakness that we leverage to develop a new attack---the proxy harvesting attack---which enables malicious clients to collect information about a large number of proxies before launching a DDoS attack. We show that current solutions are vulnerable to this attack, and propose PROTAG---a moving target defense technique consisting in periodically and proactively replacing one or more proxies and remapping clients to proxies. Our primary goal is to disrupt the attacker's reconnaissance effort. Additionally, to mitigate ongoing attacks, we develop a new client-to-proxy assignment strategy to isolate compromised clients, thereby reducing the impact of subsequent attacks. We validate our approach both theoretically and through simulation, and show that the proposed solution can effectively limit the number of proxies an attacker can discover and isolate malicious clients.