| Increasingly high volume and variety of Internet attacks are encountered every day due to attackers' strong financial motivation. Most of such malicious activities, such as Spamming, Phishing, Distributed-Denial-of-Service (DDoS), etc., involve numerous compromised computers dispersed over geographically diverse networks. As a result, Internet attackers often have strong incentives to compromise as many computers as possible around the globe. Clearly, such compromised computers pose a significant threat both to the networks they reside in and to all Internet users. Therefore, it is in the best interest of an organization to detect compromised computers and malicious activities within its network.In this thesis, we consider network-based misuse detection in an enterprise network, which is defined as the collection of all network components including computers, servers, routers, subnets, etc., under the jurisdiction of a single organization. We particularly focus on three different malicious activity detection problems The first problem is detection of relay nodes, which are often used by attackers to conceal their identities. For this problem, we present an online algorithm, which efficiently detects relay nodes in a network. Second, we consider online detection of correlated network flows. The algorithm we propose for this problem can be used in attack source attribution when attackers hide behind a series of relay nodes. Finally, completely different from the first two, the third problem we consider in this thesis is the detection of the members of a Peer-to-Peer (P2P) botnet in an enterprise network. For this problem, we present a simple graph based algorithm which identifies additional P2P bots in a network once a single bot exposes itself by sending spam or perform a network scan, etc.In general, to detect a malicious activity, a network-based misuse detection scheme monitors network traffic for certain network traffic patterns which emerge due to that particular malicious activity. One of the biggest challenges in this process is that, high volumes of network traffic corresponding to thousands of computers has to be monitored efficiently. Therefore, algorithms employed by network-based misuse detection schemes have to be simple and fast. Hence, in this thesis, we place a great emphasis on computational efficiency and scalability. |
