Cost-sensitive performance of probability estimation-based classifiers: Analysis and practice

Many schemes for classification are built using probability estimators that estimate the probability that a sample belongs to each of a given set of classes. For tasks that are cost-sensitive, in the sense that different classification errors carry different costs, probability estimation-based classifiers provide an elegant means of incorporating misclassification costs into the design of algorithms. Very often these probability estimators rely on both assumptions and simplifications that lead to inaccuracies in the estimates. Nonetheless, high classification accuracy can be achieved using inaccurate estimates of probabilities. However, for cost-sensitive classification it is the expected loss rather than the expected number of errors that is of primary concern, and the inaccuracies in probability estimation can adversely impact cost-sensitive classifier performance. When classification involves just two classes, there are simple but effective schemes to reduce the effects of these inaccuracies.; I describe how inaccurate probability estimates affect cost-sensitive classification in multi-class problems. I present an extension of bias/variance decomposition to multi-class cost-sensitive classification tasks. From this I show how two-class bias reduction schemes can be extended to multi-class problems. I also explore the effects of uncertainties in misclassification costs on the expected loss and show how to improve classifier performance when the exact misclassification costs are not known at training time. The schemes presented here improve classification for both two-class and multi-class tasks.