New Soft Margin Algorithm Adjusting The Weightd Of Weak Classifiers For AdaBoost

AdaBoost is the most representative algorithm of Boosting family, proposed byFreund and Schapire in 1995. The basic idea is to linearly combine a series of weakclassifiers by weighted voting selection to form a strong classifier. AdaBoost as anensemble learning method, based on PAC learning theory, has demonstrated itsexcellent characteristics in many application areas. The subject of this article is tostudy how to adjust the weights of weak classifiers to improve the generalizationperformance of AdaBoost.We have accomplished the following work.On the one hand, the paper describes the AdaBoost algorithm model and the ideaof the AdaBoost algorithm. We analyze the generalization error of AdaBoost algorithm,training error and its convergence, and the relationship between generalization error andthe margin. In order to further analyze the relationship between the generalization errorand margin, we introduce two adjustment AdaBoost algorithms based on margin:Arc-GV, which is based on the maximization of the minimum margin, andAdaBoost-QP, which is based on the optimization of margin distribution. We comparegeneralization errors, the minimum margins and the margin distributions of AdaBoost,Arc-GV and AdaBoost-QP experimentally, and show the importance of the margindistribution for improving the generalization performance.On the other hand, experimental results show that AdaBoost clearly overfits in thehigher noise regime, for example, the AdaBoost-QP algorithm uses optimizing margindistribution to adjust the weights of weak classifiers, but experimental results showthat the AdaBoost-QP algorithm sometimes can reach a worse generalizationperformance because the bone samples are emphasized too much. In this paper, inorder to overcome the sensitivity to noise caused by the AdaBoost algorithm, weanalyze the relationship between hard margin and overfitting, and propose'softmargin'which is designed by adding slack variables to the sample margin to relievethe effect of hard margin for noise regime classification, and propose two newAdaBoost-QP algorithms which are based on optimizing soft margindistribution-AdaBoost-QPKL,AdaBoost-QPnorm2 . Experimental results show that afteradjustment generalization performance of the two new algorithms has been effectivelyimproved.