Ensemble Learning Based On Support Vector Machines

Support vector machine is a new machine learning method based on the statistical learning theory. Because of its good generalization performance, support vector machine has been successfully applied in a variety of fields. However, there are some defects with support vector machine during the process of application. First, approximation algorithms are adopted to reduce the time and space complexities in solving the optimization problem. Second, usually the choice of kernel function depends on one's experience and the choice of parameters is done by cross validation. Though the optimality of them has no guarantee, there is no better solution so far. Third, support vector machine is in nature a tool for binary-class classification and should be extended in order to solve multi-class problems. Unfortunately, either by combining several binary support vector machines or by considering all classes in one optimization problem, the classification performance does not improve as much as in the binary classification; moreover, some methods are too hard to be implemented. These defects degrade the stability and generalization performance of support vector machine.By training and combining some accurate and diverse classifiers, ensemble learning provides a novel approach for improving the generalization performance of classification systems. In recent ten years, ensemble learning has become a main research topic in the field of machine learning. Now, the research at home and abroad of ensemble learning based on neural networks and decision trees have made great progress, while the research on support vector machine ensemble starts relatively late and needs much further studies. This dissertation focuses on developing effective ensemble learning methods with support vector machine and the main contributions are presented as follows:1) First the principle and algorithms of support vector machine as well as the extension methods for multi-class classification are introduced. The general methods of ensemble learning are summarized in detail from both the construction and combination aspects of base classifiers. The current developments of ensemble learning research on support vector machine at home and abroad are reviewed2) An ensemble learning method based on attribute reduction is proposed. Attribute reduction methods in the rough set theory can be used as a preprocessing technique of redundant data for learning algorithms. However, it may reduce the classification performance of learning algorithms in many cases due to the influence of data noise and discretization. Reduction of a decision table with redundant attributes can produce more than one different reduct of attributes. The reducts usually have relative good classification capabilities and are different from each other. Therefore, the reducts can be utilized to construct support vector machine ensembles. Reduction based ensemble can utilize effectively the complementary or redundant information in the training data for fusion classification and overcome the harmful influence of attribute reduction on the classification performance of support vector machine.3) A construction method of base classifiers based on discretization of attributes is proposed. Three possible implementation strategies are pointed out: choosing cuts randomly; adopting some discretization algorithm and choosing different numbers of cuts; or adopting several discretization methods. In this paper, the first strategy is used to construct support vector machine ensemble based on the RSBRA discretization method. Aiming at the disadvantage of RSBRA that it may excessively degrade the classification performance of support vector machine, the level of consistency, which is coined from the rough set theory, is introduced to modify RSBRA so as to preserve enough information for classification. Afterwards, an ensemble learning method based on the modified RSBRA discretization method is proposed.4) In present ensemble learning methods based on search techniques, measures of performance are needed to evaluate the base classifier. Nevertheless, these measures either are hard to be adjusted to make a reasonable tradeoff between accuracy and diversity, or can not reflect directly the generalization performance of an ensemble. Aiming at this problem, we propose a direct genetic ensemble method which searches for a good ensemble in the ensemble space by genetic algorithms. The presented method can be implemented to produce selective ensembles of classifiers readily. The study shows that selective ensembles gain better classification performance than traditional ensemble learning methods such as Bagging and Adaboost by combining less classifier.5) The combination structures of classifiers in multi-class classification problems are studied. A simplified structure is proposed to overcome the defects of existent structures. Based on the structure, the measurement-level combination methods of classifiers based on the evidence theory are studied. In the evidence theory method, the basic probability assignment functions are defined by utilizing the posterior outputs and predictive accuracies of support vector machines and then combined by some rule. In particular, when the one-against-one method is used for multi-class extension, the evidence may conflict heavily and the classic Dempster combination rule is not applicable. Therefore, we propose a new evidence combination rule based on the thought that the conflicting information is partly valuable. The valuable part is determined according to the globe effectiveness of the evidence and then distributed among the focus elements according to the weighted sum of basic probability assignments. The rule tackles the problem of evidence conflicting effectively.