| Benchmarking has been used to compare the performance of a variety of technologies, including computer systems, information retrieval systems, and database management systems. In these and other research areas, benchmarking has caused the discipline to make great strides. Until now, research disciplines have enjoyed these benefits without a good understanding of how they were achieved. In this dissertation, I present a theory of benchmarking to account for these effects. This theory were developed by examining case histories of successful benchmarks in computer science, my own experience with community-wide tool evaluations in software reverse engineering, and the literature from philosophy of science. According to the theory, it is the tight relationship between a benchmark and the scientific paradigm of a discipline that is responsible for the leap forward. A scientific paradigm, as described by Thomas S. Kuhn, is the dominant view of a science consisting of explicit technical facts and implicit rules of conduct. A benchmark operationalises a scientific paradigm; a benchmark is a statement of the discipline's research goals and it emerges through a synergistic process of technical knowledge and social consensus proceeding in tandem. The theory of benchmarking is validated empirically against extant benchmarks and analytically against a hierarchical set of criteria.; An important implication of the theory is that benchmarking can be used to cause a scientific discipline to make advances. To this end, a process model has been developed as a guide for others wishing to undertake a benchmarking project within their research communities. Also provided are a Benchmarking Readiness Assessment and criteria for evaluating the benchmarking process. The application of the theory and these process model is illustrated using two benchmarks that I have developed with the software reverse engineering community. These are the xfig benchmark for program comprehension tools and C++ Extractor Test Suite (CppETS), a benchmark for comparing fact extractors for the C++ programming language. |
