Statistical methods for performance evaluation and their applications

Statistical performance evaluation has many applications. In these applications, many alternative solutions or hypotheses exist and the ones performing the best in terms of predetermined measurements are sought. The performance measures of hypotheses are numerical numbers and have to be obtained based on examples and may contain noise. In addition, due to the time and resource constraints in real applications, it is often impractical or even impossible to evaluate all hypotheses. Thus, statistical metrics are used to evaluate the performance of hypotheses efficiently using a limited number of examples and tests. There are many statistical metrics available and their results depends on many factors, such as the number of test cases, whether or not the performance measurements are noisy, and the distribution of performance measurements of the hypotheses. Selecting the most appropriate statistical metrics is a challenging task.; In this dissertation, we propose a general framework for statistical performance evaluation. The framework incorporates various statistical metrics and automatically selects the most appropriate one based on the characteristics of the application problem. We have identified the following important problem characteristics: the number of hypotheses, the size of sample data for each hypothesis, the distribution of performance measurements, and the distribution of noise in performance measurements. Then, we apply statistical performance evaluation methods to four applications: evaluation of search engine performance on the Web, analysis and improvement of HITS-based document ranking algorithms, optimization design of filter banks for image compression, and optimization design of filter banks for signal denoising. In the first application, we apply statistical methods to evaluate the precision of search engines. We have performed extensive experiments using real search engines on the Web and obtained promising results. In the second application, we statistically analyze the performance of the combination of HITS-based algorithms and relevance scoring methods, and develop a adaptive weighting method which achieves better results without any content analysis. In the third application, we develop an optimization-based approach to design biorthogonal filter banks for image compression, in which statistical performance evaluation methods are used to select the solutions that are more generalizable to other images unseen in the optimization design stage. Similarly, in the fourth application, we develop an optimization-based method for designing orthonormal filter banks for signal denoising and apply statistical performance evaluation methods in selecting more generalizable solutions. In these two applications, our methods have obtained filter banks that perform better than the benchmark existing filter banks.