Security Policy Definition and Enforcement in Distributed Systems

Security in computer systems is concerned with protecting resources from unauthorized access while ensuring legitimate requests can be satisfied all the time. The recent growth of computer systems both in scale and complexity poses tremendous management challenges. Policy-based systems management is a very promising solution in this scenario. It allows the separation of the rules that govern the behavior choices of a system from the provided functionality, and can be adapted to handle a large number of system elements. In the past two decades there have been many advances in the field of policy research. Although existing solutions in centralized systems are well-established, they do not work nearly as well in distributed environments because of scalability, network partitions, and the heterogeneity of the endpoints.;This dissertation contributes to this endeavor by proposing three novel techniques to address the problem of security policy definition and enforcement in large-scale distributed systems. To correctly enforce service and security requirements from users who have no intimate knowledge of the underlying systems, we introduce the first distributed policy refinement solution that translates high-level policies into low-level implementable rules, for which the syntax and semantics can be fully interpreted by individual enforcement points. Taking advantage of both the centralized and end-to-end enforcement approaches, we propose a novel policy algebra framework for policy delegation, composition and analysis. As a concrete instantiation of policy delegation enabled by the algebraic framework, we invent a novel firewall system, called ROFL (ROuting as the firewall layer), that implements packet filtering using the underlying routing techniques. ROFL implements a form of ubiquitous enforcement, and is able to drop malicious packets closer to their origins to save transmission bandwidth and battery power, especially for resource-limited devices in mobile ad hoc networks (MANET). The correctness and consistency of ROFL can be verified using policy algebra. It provides formalisms to address the complexity of distributed environments, increase assurance and show how to tune tradeoffs and improve security with ubiquitous enforcement.;To demonstrate the effectiveness and efficiency of ROFL as a high-performance firewall mechanism, we analyze its performance quantitatively and conduct experiments in a simulated environment with two ad-hoc routing protocols. Empirical study shows that the increase in traffic for handling ROFL routing messages is more than outweighed by the savings by early drops of unwanted traffic.