| The problem of gaining assurance on the correctness of fault-tolerant protocol implementations is difficult in the current state of the art. Techniques based on formal verification and analytical modeling are either not applicable, or not accurate enough for complex highly-available systems. The most viable approaches test implementations by subjecting them to experiments that include both normal workloads and failures injected into the execution, to test the protocol's behavior under the failure scenarios it is supposed to tolerate. In this dissertation, we identify several important shortcomings of existing approaches, and show that some of them are impossible to overcome in practical distributed systems.; We introduce centralized failure injection, a novel paradigm for testing implementations of real-time and fault-tolerant communication protocols. Our approach is based on running all processes in a single address space while providing the appearance of a truly distributed execution. By a careful choice of scheduling and virtual-time management algorithms, our approach overcomes the shortcomings of previously existing solutions, while providing omniscient observation capabilities that are impossible to implement in a distributed testing platform. Unlike previous simulation environments, we can accurately represent the real-time characteristics of the real system being simulated.; We prove that our scheme can emulate exactly the set of runs feasible in the real system, while providing an unprecedented degree of control over the progress of test experiments. Furthermore, properties in protocol specifications can be accurately evaluated during experiments. By discussing several case studies, we show how our paradigm can be used to test various protocols that assume widely different system models and failure semantics. In addition to testing the correctness of implementations, we present our results in performance evaluation, where our simulator was able to provide very accurate predictions of the real system's performance. |
