Performance measures of machine learning

This thesis investigates some fundamental issues of performance measures of machine learning.;We first formally propose criteria to compare performance measures. These criteria focus on studying the consistency relationship between two measures, and whether one measure has more discriminatory power than the other. Based on the proposed criteria, we theoretically and empirically compare two most popular measures: accuracy and AUC (Area Under the ROC Curve). We show that AUC is statistically consistent and more discriminant than accuracy, which indicates that AUC should be preferred over accuracy in evaluating learning algorithms. We also compare ranking measures and give a preference order to use these measures in comparing ranking performance.;Based on the comparison criteria, we propose two general approaches to construct new measures from existing measures. We formally prove that the new measures are consistent and more discriminant than the existing ones. We also compare the learning models of artificial neural networks trained with the newly constructed measures and existing measures. The experiments show that the model trained with the newly constructed measure outperforms the models trained with the existing measures.;Finally, we explore model selection tasks using measures. We show that generally we should use different measures as model selection goal and evaluation measures. We show that a measure's model selection ability is stable to model selection goal and class distributions. We find that some measures perform better than others in the model selection tasks.;Performance measures (or evaluation measures) play important roles in machine learning. They are not only used as the the criteria to evaluate learning algorithms, but also used as the heuristics to construct learning models. However, little work has been done to thoroughly explore the characteristics of performance measures.;In summary, this thesis addresses several fundamental issues of machine learning measures. The research results are very useful in real world applications. It provides the guidance on how to select suitable measures to evaluate learning algorithms. Furthermore, it also presents general approaches to construct new measures efficiently and effectively, which provides new approaches in building learning models.;Keywords. performance measures, threshold measures, ranking measures, probability-based measures, comparison criteria, constructing measures, model selection.