Study For Confidence Interval Of AUC Measure Based On Cross-Validation

In the research of statistical machine learning,algorithm performance measure runs through the whole process of model evaluation and model selection.Therefore,the research of algorithm performance measure is a very important research direction.In general,the commonly used algorithm performance measures mainly include algorithm performance measures based on error rate,algorithm performance measures based on confusion matrix,and algorithm performance measures based on statistical test.And the algorithm performance measures based on confusion matrix is more widely used in the application.Algorithm performance measures based on confusion matrix include precision,recall,F₁ measure,sensitivity,specificity and so on.However,these measures are given either based on a single threshold,which is susceptible to category imbalance and different classification misclassification costs,or there may be a contradictory situation where one measure is high and the other is low.For this reason,ROC curves that are insensitive to classification changes and AUC measure based on ROC curve area are proposed.In view of the superiority of AUC measure,which is widely used in various fields,this paper focuses on confidence interval of AUC measure.In particular,note that the research of AUC measure always only consider its point estimation problem.This will result in instability of the results because of ignoring the variance informance.At the same time,Wilcoxon-Mann-Whitney test is used to estimate AUC measure in most literatures.However,Wilcoxon-Mann-Whitney test is a non-parametric statistical method that does not assume the data distribution,which may easily result in inaccurate results.Therefore,this paper will study the confidence interval of AUC measure by considering the approximate distribution of AUC measure.The specific work and innovation points are as follows:1.The AUC measures based on different cross-validation techniques(K-fold cross-validation,blocked 3×2 cross-validation)are proposed in two situations of a given threshold and multiple thresholds.This paper fully considers the original definition of AUC measure and the cross-validation technique.By increasing the number of thresholds one by one,the AUC measures based on different cross-validation techniques are given in two situations of a given threshold and multiple thresholds.The accuracy and rationality of results are further verified by the experimental results.2.In the two situations of a given threshold and multiple thresholds,the confidence intervals of AUC measure based on the cross-validated Beta distribution are respectively proposed.The traditional confidence intervals of AUC measure are usually constructed based on normal assumptions,such as the confidence interval of AUC measure based on the cross-validated t distribution,the confidence interval of AUC measure based on the corrected cross-validated t distribution.Obviously,these confidence intervals are symmetric.However,through the theoretical analysis of AUC measure,the real distribution of AUC measure is a skewed distribution on the interval(0,1).In this case,it is obviously inappropriate to simply use the symmetric distribution to approximate the distribution of AUC measure.They always exhibit low degrees of confidence or long interval lengths.In addition,the confidence interval constructed by symmetric distribution is likely to exceed the interval(0,1),which may easily result in incorrect statistical inference results,which is also demonstrated by the experimental results.Therefore,this paper proposes a new asymmetrical confidence interval of AUC measure based on the cross-validated Beta distribution in the two situations of a given threshold and multiple thresholds.A large number of simulated and real data experiments verify that the proposed method has shorter interval length and higher degrees of confidence than the symmetric confidence interval of AUC measure based on the traditional cross-validated t distribution and corrected cross-validated t-distribution.