The Study For The Comparison Of Classification Algorithms Based On Balanced 5×2 Cross-Validation

In statistical machine learning, when proposing a new algorithm, we usually need to compare its performance with the previous algorithms and use statistical significance test to yield the conclusion that the improvement of the new algorithm’s performance is significant or not. In this paper, a new balanced 5x2 cross validation is proposed by combining blocked 3×2 and random 5×2 cross validation, which having identical number of overlapped samples for any two groups training sets (test sets). Then, the corresponding balanced 5x2 cross validated F test is given, and the properties of the test statistic and the degrees of freedom (DOF) for the test are theoretically analyzed. For single and multiple datasets, the perfor-mances of two classification algorithms are compared based on balanced 5×2 cross validated test.Combined 5×2 cross validated F test based on five replications of 2 fold cross validation is commonly used in comparing the performances of algorithms on single dataset. However, the reusing of the same data in 5x2 cross validation causes the real DOF of the test to be lower than the theoretical value. This easily leads to the test suffering from high type I and type II errors. Noting that the group-in and group-out correlations in random 5x2 cross validation are related to the number of overlapped samples between training sets, but random partitions for 5x2 cross validation result in the different number of overlapped samples between training sets as well as difficulty in analyzing the test statistic. Based on this, this paper puts forward a balanced 5x2 cross validated F test. Balanced partitions make the group-in and group-out correlations become identical and the theoretical analysis for the degree of freedom of test is conducted. Theoretical analysis finds that the group-out correlation coefficient affects the change in the degrees of freedom of the test and a calibrated balanced 5x2 cross validated F test by calibrating DOF is put forward. In addition, similar to blocked 3x2 cross validated t test in literature a balanced 5x2 cross validated t test is also proposed. Simulated experiments show that balanced 5x2 cross validated t test and balanced 5x2 cross validated F test(the DOF is (7,5)) have better performance in comparing two classification algorithms. Furthermore, balanced 5x2 cross validation and its tests can be extended to balanced mx2 cross validation. Also the properties of the test statistic and the DOF for the F test are theoretically analyzed. However, the selection of m=5 is appropriate through the theoretical analysis for variance and the simulation for power function.However, with the same test the results may be different on different datasets. This paper considers the comparisons of two algorithms over multiple datasets such that the results of the test have better generalization properties. Thus, we apply balanced 5x2 cross validation to the existing paired t test. Also relative paired t test and Wilcoxon signed ranks sum test. The relative paired t tests based on balanced 5×2 cross validation and 10 fold cross validation are shown better performance by simulated experiments.