| This thesis introduces a novel approach to performance evaluation, called application-specific benchmarking, and presents techniques for designing and constructing meaningful benchmarks.; A traditional benchmark usually includes a fixed set of programs that are run on different systems to produce a single figure of merit, which is then used to rank system performance. This approach often overlooks the relevance between the benchmark programs and the real applications they are supposed to represent. When the behavior of the benchmark programs do not match those of the intended application, the benchmark scores are uninformative, and sometimes can be misleading. Furthermore, with the rapid pace of application development, it is impractical to create a new standard benchmark whenever a new “killer” application emerges.; The application-specific benchmarking approach incorporates characteristics of the application of interest into the benchmarking process, yielding performance metrics that reflect the expected behavior of a particular application across a range of different platforms. It also allows benchmarks to evolve with applications and consequently the benchmarks are always up-to-date.; This thesis applies the application-specific benchmarking methodology to a variety of domains covering Java Virtual Machines, garbage collection, and operating system. The result is a collection of benchmark suites that comprise the HBench framework, including HBench:Java, HBench:JGC, and HBench:OS.; This thesis demonstrates HBench's superiority in predicting application performance over a more conventional benchmarking approach. It is shown that HBench:Java is able to correctly predict the rank of running times of three commercial applications on a variety of Java Virtual Machine implementations. In the realm of garbage collection, the predicted garbage collection times for a stop-the-world, mark-sweep garbage collector closely match the actual times. In the domain of operating system, it is demonstrated that HBench:OS can be used to analyze performance bottlenecks and to predict performance gains resulting from a common optimization. |
