A Study Of Classifier Ensemble Methods Based On Greedy Optimization And Projection Transformation

With the continuous development of computer science, the require of theperformance of machine learning become increasingly demanding, while more complexissues to be addressed, which indicates that a single learner model can not satisfy forfully the demand for higher performance. Ensemble learning has been proposed as anew machine learning algorithm, and it can improve the performance of a single learnerby combining multiple learners. Because ensemble learning can improve significantlythe performance of a single learner, it becomes a hot research in the field of machinelearning from the1990s. Moreover, classifier ensemble is considered as a typicalapplication of ensemble learning in supervised learning, which also improves theperformance of a single classifier by combining the outputs of multiple classifiers.Currently, classifier ensemble has been successfully applied to many practical problems,such as face detection, remote sensing data classification, medical image processing,network data processing and so on.For classifier ensemble, diversity among individual classifiers and accuracy ofindividual classifiers themselves are two important factors which decide the ensemblegeneralization error. In short, both enhancing diversity among individuals andincreasing accuracy of individuals can decrease the ensemble generalization error.However, the researches illustrate that enhancing diversity is at the cost of reducingaccuracy of individuals in an ensemble system. It means that balancing diversity andaccuracy is not only a starting point but also a difficulty for constructing an ensemblemethod. In order to construct a better ensemble system, this paper around diversity,accuracy and ensemble error to design methods based on how to combine the outputs ofindividual classifiers in an ensemble system, and have several proposed classifierensemble algorithms, which are shown as follows:1. In order to improve the performance of an ensemble by balancing diversity andaccuracy, a method has been proposed that selective classifier ensemble via greedyoptimization. In this method, diversity and accuracy are considered at the same time,and an optimal combination of individual classifiers is sought by using matching pursuitalgorithm in an ensemble system. Inspired by matching pursuit, the outputs ofindividual classifiers are considered as the basic functions, and the true labels are as thetarget function. Then the optimal combination is obtained by minimizing the residualbetween the target function and the combination of basic functions. According to thetheoretical analysis, it demonstrates that the proposed method can eliminate some similar individual classifiers by giving them zero coefficients in each iteration and retainthe individual classifier with better performance in the initial iteration, even giving it ahigher coefficient. Experimental results illustrate that the proposed method can obtainbetter classification performance compared with other ensemble methods.2. On account of the difficulty of balancing diversity and accuracy, a selectiveensemble method based on transformation of classifiers has been proposed. In order tobalancing diversity and accuracy effectively, the proposed method is constructedrespectively from diversity and accuracy. Firstly, for enhancing diversity of an ensemblesystem, individual classifiers are transformed by a linear transformation method, and aset of new individuals is obtained. Then, for ensuring the performance of the ensemblewith transformation, a selective strategy is designed based on two rules that measure theperformance of a classifier. Finally, the selected transformed classifiers are combined.Experimental results illustrate that the proposed method balances efficiently diversityand accuracy by transforming and selecting classifiers and obtains better performancethan other ensemble methods.3. In order to avoid the difficulty derived from designing an ensemble methodbased on diversity and accuracy, a new method has been proposed that weightedclassifier ensemble based on quadratic form. Based on the final intension of classifierensemble, the proposed method makes an analysis of ensemble error instead of diversityand accuracy. Then an optimal weight vector is obtained by minimizing the ensembleerror directly. In the proposed method, an approximation error with two constraintconditions is given and considered as the target function, instead of the general form ofensemble error, and then the minimization of the approximation error is converted to themaximization of a quadratic form by introducing a given weight vector. According tothe theoretical analysis, it demonstrates that when the value of the quadratic form isbigger, the ensemble error obtained by the weight vector corresponding to that value ofquadratic form is lower. Experimental results illustrate the classification performance ofthe proposed method is superior to other compared ensemble methods.4. Based on the frame of weighted classifier ensemble, a new ensemble method hasbeen proposed that weighted classifier ensemble based on linear transformation.Inspired by linear transformation, the process of weight classifier ensemble isconsidered as a process of linear combination of individual classifiers. Hence, theproposed method introduces linear transformation into ensemble learning, and it seeksthe optimal weight vector of individuals by linear transformation methods. But the starting point of linear transformation is very different from ensemble learning, so theproposed method employs the true labels to construct a correlation matrix representingall individuals, instead of the mean used in linear transformation. Based on thecorrelation matrix, an optimization target function is made to seek the optimal weightvector. According to the theoretical analysis, it demonstrates that the value of the targetfunction is equivalent to the weight sum of accuracy of all individuals in an ensemble.When the target function is bigger, the accuracy of the whole ensemble system is bigger.Experimental results illustrate the proposed method obtains better classificationperformance compared with other ensemble methods.5. In order to improve the performance of an ensemble system, an ensemble methodhas been proposed that weighted classifier ensemble based on the decomposition of0-1matrix. The proposed method uses a0-1matrix to represent individual classifier ofan ensemble, and then an optimal weight vector of individual classifiers is obtained bysingular value decomposition for0-1matrix. According to the theoretical analysis, itdemonstrates that when the singular value of0-1matrix is higher, the ensemble errorobtained based on the weight vector corresponding to that singular value is lower.Experimental results illustrate that the proposed method is simpler and more effectivecompared with other ensemble methods.6. In order to enhance diversity of an ensemble, an isomerous classifier ensemblemethod has been proposed. The proposed method applies rotation forest strategy tocombine two different classifier models. In the proposed method, firstly, the rotationforest strategy is used to transform for the original sample set, and new sample sets areobtained. Then, the classifiers with high classification performance are selected as thebasic classifier model by the specified percentage. Finally, the outputs obtained basedon the selected model and the transformed samples sets are combined. Experimentalresults indicate that the proposed method improves the classification accuracy comparedwith homogeneous models, while achieves effectively the complementarity of accuracyand speed.