| In studies separated by decades, operator mistakes have been identified as a significant source of unavailability in computer systems. Such mistakes can range from static misconfiguration to physical misplacement of wires and machines. Detecting and repairing these mistakes can often be time consuming and for many of today's Internet services, unavailability results in significant loss of revenue and/or clients.;In this dissertation we present a series of tools that assist those who are charged with designing and creating the infrastructure that support those Internet services to mitigate the results of operator mistakes. Specifically, we first propose an assertion based language, A, that is a formalized specification of correct behavior and can be used to bolster system understanding, as well as help to flag operator mistakes in a distributed system. We look at examples of these mistakes, their effects and manifestations in both an academic environment and in real-world applications. With a good understanding of these mistakes and behaviors, we design a process to validate operator actions called Model Based Validation. We then explore methods to simplify the assertion writing process using machine learning techniques. With such a large attribute space and such small data sets, we investigate a variety of optimizations including a refinement loop and various filtering algorithms. |
