Regularization methods for support vector machines

Support Vector Machine (SVM) in a broad sense refers to the learning approaches that only a subset of data/feature vectors are used to support the model construction while others can be safely excluded. The present thesis sets forth one of its main themes on new regularization methods that can lead to SVM formulations. It also deals with efficiently tracing the solution paths for a broad family of SVMs as their regularization parameters take every possible value.;The thesis devotes a significant portion to new regularization methods and their applicable domains. For ensemble learning, the thesis proposes a set of regularized linear combination approaches, which outperform many ensemble methods that are heuristic in nature and do not have a well-defined objective function. Consequently as driven by the different effects of sparse weights on regression and classification ensembles, the thesis further contributes two extended studies on regularization methods. First, a zero-norm support vector regression is proposed to progressively reduce the number of nonzero weights; second, an entropy penalty criterion together with large margin constraints is developed to conservatively keep the weights as uniformly as possible in improving empirical performance. Due to the different even contrary nature of these two extensions, they are applied to different scenarios. The zero-norm approach is tested on real regression data. It demonstrates the favorable properties of selecting a compact set of feature coefficients and tolerating small fitting errors. While the entropy approach is used to optimize the majority voting of the bagging approach in classification ensemble learning. Both the extensions achieve performance improvement for the original tasks. However, they are limited to conventional classification and regression tasks, thus inapt at the new learning scenarios where unlabelled data are plenty and the data show intrinsic smoothness. To overcome this limitation, the thesis designs a new regularized objective function that takes into account the manifold consistency over all labelled and unlabelled data through a graph operator. It can achieve significant performance gain on pattern recognition tasks by only requiring a small training set.;The second part of the thesis is centered on regularization parameters, by focusing on tracing the piecewise linear solution paths for a broad family of SVMs. It is a work that can help parameter sensitivity analysis and model selection. The thesis presents a unified approach that can trace the solution paths more systematically and robustly. Compared with existing approaches, this method is in a principled way and generalizes to many SVM variants. In addition, a thorough study on the degenerated cases in tracing solution paths is conducted, and efficient methods to handle degenerated cases are proposed. More importantly, the approach has drawn a novel relation with classic Mean-Variance portfolio optimization techniques and is of potential to revitalize the parametric quadratic programming study.