Statistical Inference Of Classification Learning Algorithm Based On Blocked3×2Cross-Validation

The model performance evaluation is a key step in statistical machine learning. Its quality directly impact on many other aspects of machine learning. The generalization error is the most commonly used and most important model performance evaluation metric. At present, generalization error is often estimated by various cross-validation methods. We study the new proposed blocked3×2cross-validation from aspects of model selection、 variance analysis、variance estimation and hypothesis testing for classification learning algorithm. And good conclusions is obtained.Blocked3×2cross-validation have two major characteristics:smaller fold and balanced cutting data. So we first comprehensively analyse cross-validation methods from the selection of fold and the cutting of data. We judged that blocked3×2cross-validation has superiority in this two aspects based on the existing research results.We apply blocked3×2cross-validation to model selection task of classification learning algorithm. Considering the characteristics of blocked3×2cross-validation and the influencing factors of the performance for cross-validation model selection method. We assert that the blocked3×2cross-validation is superior to the commonly used5-fold and10-fold cross-validation. When the feature values is continuous, the experimental results show that blocked3×2cross-validation has bigger probability of selecting the true model than5-fold and10-fold cross-validation, and the most up to25percentages. There are individual exceptions when feature values is discrete. But the advantage of10-fold cross-validation is not obvious than blocked3×2cross-validation. Compared with the former, the latter has a great advantage in computational complexity. So we think blocked3×2cross-validation is more suitable for model selection task of classification learning algorithm than5-fold and10-fold cross-validation.When comparing the performance of classification learning algorithm, we should perform the hypothesis testing for generalization error of algorithm. But the hypothesis testing need a effective variance estimation. So we first analyse the variance of blocked3X2cross-validation from theory, and get its structure diagram. Based on this analysis, we propose a conservative estimate about the variance of blocked3X2cross-validation and use it for hypothesis testing. The experimental results show that blocked3X2cross-validation t-test is better than others. In the other words, it has smaller type I error and bigger power.Many problems in natural language processing can be seen as a classification task. The commonly used evaluation metrics about these algorithms are Precision、Recall and F-score. As long as every accurately or approximately obeys normal distribution. The above conclusions about generalisation error can be used for statistical inference based on these metrics. The previous results show that Precision and Recall obey beta distribution, and beta distribution is approximately equal to normal distribution when its two parameter values are bigger. According to this result and the relation of F-score and Precision、Recall, we prove that the F-score also approximately obeys normal distribution.