Research On Selective Ensemble Learning Algorithm Based On Margin And Confidence

Ensemble learning has been a hot topic in machine learning domain for more than twenty years. The generalization performance can be significantly improved by combin-ing a set of base learners. However, it entails large memory requirement and slow speed of prediction as the number of base learners increases. On the other hand, it can reduce the memory requirement and increase the prediction speed via selecting a fraction of the base learners. More importantly, it has been theoretically and empirically proved that selective ensemble can improve the generalization ability of the original system. Thus selective ensemble learning is one of the important problems in ensemble learning domain.This dissertation studies the design and application of selective ensemble learning algorithms based on margin and confidence. In particular, the main work can be summa-rized as follows:1. A new static selective ensemble learning algorithm, named DRMF, is proposed from the viewpoint of margin. It aims to acquire good generalization ability by improving the margin distribution over the training sets. Double Rotation and Margin Based Pruning are introduced to construct DRMF, where the former can improve the margin distribution by train a set of diverse base classifiers, and the latter is utilized to select a fraction of the base classifiers by minimizing the classification loss in terms of margin. The effect of different parameters to the performance of DRMF is discussed in detail and its robustness is also analyzed. Experimental comparison validates the classification performance of DRMF, and its success is explained from the viewpoints of margin and diversity.2. The definition of margin is generalized based on the classification confidences of base classifiers. Then the role of classification confidence is analyzed in detail in selec-tive ensemble. In particular, the difference is shown when learning the weights of base classifiers based on different margins, and the necessity of incorporating the classification confidence into margin is also explained. After that, some experiments are conducted to explore how to utilize the classification confidences in selective ensemble based on or-dered aggregation technique and a new algorithm, named EP-CC, is proposed. Finally, the effect of classification confidence to EP-CC is explored and the success of the proposed algorithm is explained from different viewpoints. 3. A new dynamic selective ensemble algorithm, named DES-NC, is proposed by optimizing the number of the selected base classifiers. It is worth noting that the numbers of the selected base classifiers are the same for different samples, but the selected classi-fiers are different. Firstly, the motivation of using classification confidence is explained and the process of selecting the base classifiers is introduced in detail. Then some ex-periments are conducted to verify the rationality of the proposed algorithm. Finally, the success of DES-NC is explained from the view point of optimizing the margin distribu-tion.4. A new dynamic selective ensemble algorithm, named DES-TV, is proposed based on the threshold value. Compared with DES-NC, the selection strategy for DES-TV is more flexible and the numbers of the selected base classifiers are not required to be the same for different samples. Some experiments are conducted to show the variation of classification accuracies with different threshold values, and analyze the classification per-formances of DES-TV with different loss functions. Besides, a new method is proposed to compute the confidence for the output of regressor. Then DES-TV can be utilized in s-elective regression ensemble, and its prediction performance is validated in the prediction of wind speed.