An analysis of penetration vulnerabilities in system software

The effect of software flaws has long been considered to be an “all-or-nothing” property of system security. That is, all software flaws that allow an unprivileged user to obtain system privilege were considered to make a system insecure and no other flaws were considered to be a serious security threat. Although there is agreement that the various flaws that do not yield system privilege may vary in severity, determining the different effects of these flaws has always been an ad hoc, unsystematic procedure. As a consequence quantitative measures of these effects have long been believed to be unobtainable.; In this dissertation we show that the effects of flaws that do not yield system privilege can vary in the severity of the resulting security exposures, and that the degrees of severity can be measured and compared. Also, since computer systems used for different applications implement different security policies, the same exact flaw can have different repercussions relative to these policies.; We provide two separate metrics for quantifying and comparing the severity of various security flaws and the resultant varied levels of security. These metrics are based on several quantifiable properties that define the severity of a software flaw. We use examples from a theory of penetration resistant systems to illustrate our metrics, and show that any violations of the properties of penetration resistance can be measured and compared using our metrics. In doing this we show how an existing class of tools can be employed to partially automate measurement of security flaws.; Furthermore, each flaw is associated with a region of vulnerability , which consists of the set of specified security levels such that a system operating at any of those levels is vulnerable to the flaw. Systems that are operating outside of the region of vulnerability do not need to take immediate steps to protect themselves from the associated flaw since the flaw does not cause these systems to operate below their specified level of security.