| Event correlation is a widely-used data processing methodology, and is useful for the distributed monitoring of software faults and vulnerabilities. Most existing solutions have focused on "intra-organizational" correlation; organizations typically employ privacy policies that prohibit the exchange of information outside of the organization. However, "inter-organizational" Internet-scale correlation holds promise given its potential role in both software fault maintenance and vulnerability detection.; In this thesis, I reconcile these opposing forces via the use of privacy preservation integrated into an event processing framework. I introduce the notion of event corroboration, a reduced yet flexible form of correlation, enabling collaborative verification without revealing sensitive information. The framework supports both source anonymity and data privacy, yet allows for temporal corroboration of a broad variety of data. It is designed as a lightweight collection of components to enable integration with existing COTS platforms and distributed systems. I also present an implementation: Worminator, a Collaborative Intrusion Detection System (CIDS); it is based on an earlier platform, XUES (XML Universal Event Service), an event processor used as part of an autonomic software monitoring, reconfiguration and repair platform.; XUES collected and correlated information from sensors installed in legacy systems; while it was not privacy-preserving, it laid the groundwork for Worminator by supporting event typing, the use of publish-subscribe and extensibility support via pluggable modules. In turn, Worminator is a rewrite of XUES to support privacy-preserving event types and algorithms, enabling intrusion detection alerts to be corroborated without revealing sensitive information about a contributor's identity, network or services. Worminator is implemented as a corroboration framework on top of existing IDS sensors, and can detect not only worms but also "stealthy" scans; traditional single-network sensors overlook or miss them entirely. Worminator corroborates packet metadata, packet content, and even aggregate models of network traffic.; The contributions of this thesis include the development of an event processing framework with native privacy-preserving types, the use of privacy-preserving corroboration, and the establishment of a practical deployed collaborative security system. The thesis also quantifies Worminator's effectiveness at attack detection and its privacy preservation techniques. |
