| Accurate benchmarking is an important and open research problem, made challenging because of difficulties encountered in accounting for monitoring overhead and validating measurements. By studying the performance measurement using probes inserted into the source code accessing the hardware performance counters, we develop a framework that addresses the overhead and validation problems. This framework is formulated within some existing frameworks for general software measurements. To validate a measurement we use multiple instrumentation systems to obtain measurements that can be cross checked. For each one of these systems we extend; apply and evaluate models and methodologies for overhead compensation and overall validation. The sources of errors and overhead include both the instrumentation itself as well as activities unrelated to the phenomenon under study. We validate our flow-graph based instrumentation and perturbation model on small cases using static instruction counting, basic cache theory, and interrupt vs. cycle frequencies. We validate on larger cases based on the precision in which the overall performance of a system can be estimated from its instrumented execution using the existing Linux system calls, shells and user mode hardware counter access instructions. Finally we extend, apply and evaluate for the practical FFT library (FFTW) this thesis' methodologies for overhead compensation and overall validation. |
