Defending against Internet worms

With the capability of infecting hundreds of thousands of hosts, worms represent a major threat to the Internet. While much recent research concentrates on propagation models, the defense against worms is largely an open problem. This proposal develops two defense techniques based on the behavior difference between normal hosts and worm-infected hosts.; In the first part of the dissertation, we propose a distributed anti-worm architecture (DAW) that automatically slows down or even halts the worm propagation. The basic idea comes from the observation that a worm-infected host has a much higher connection-failure rate when it scans the Internet with randomly selected addresses. This property allows DAW to set the worms apart from the normal hosts. A temporal rate-limit algorithm and a spatial rate-limit algorithm, which makes the speed of worm propagation configurable by the parameters of the defense system, is proposed. DAW is designed for an Internet service provider to provide the anti-worm service to its customers. The effectiveness of the new techniques is evaluated analytically and by simulations.; In the second part of the dissertation, we propose a defense system that is able to detect new worms that were not seen before and, moreover, capture the attack packets. It can effectively identify polymorphic worms from the normal background traffic. The system has two novel contributions. The first contribution is the design of a novel double-honeypot system, which is able to automatically detect new worms and isolate the attack traffic. The second contribution is the proposal of a new type of position-aware distribution signatures (PADS), which fit in the gap between the traditional signatures and the anomaly-based systems. We propose two algorithms based on Expectation-Maximization (EM) and Gibbs Sampling for efficient computation of PADS from polymorphic worm samples. The new signature is capable of handling certain polymorphic worms. Our experiments show that the algorithms accurately separate new variants of the MSBlaster worm from the normal-traffic background.; In the third part of the dissertation, we further investigate the multiple PADS model and propose the optimization of our iterative approaches. We propose a new way of extracting multiple PADS blocks at the same time using iterative methods such as Expectation-Maximization (EM) algorithm. To classify different polymorphic Internet worm families, we revisit the EM algorithm based on a Gaussian mixture model for each byte sequence, which is assumed to be in a feature vector space. The algorithm proposed saves the time complexity of the iterative approaches in that the extraction step can be done simultaneously.