Kernel Methods And Model Selection For Support Vector Machines

Support vector machines (SVM), born in 1990s, are a kind of excellent mathematical model. SVM constructs the predicting rules by controlling the capacity of approximating functionals and the empirical errors. SVM is widely used in many fields of machine learning, such as pattern recognition, classification, function approximation, and clustering.Generally, SVM works in feature space by kernel mapping, and this aims to acquire linear property in feature space for nonlinear problems in input space. Linearity is a desired property for pattern problems. All the dependency relations of samples are measured by kernels, and a priori are carried into SVM by kernels. So, kernel methods belong to key studies for SVM. Besides, model selection is a key step for constructing SVM. Model selection finds the optimal parameters for SVM.One-class SVM belongs to unsupervised learning machine models and its model selection has always been executed manually with shortage of criteria. This is not reasonable. In fact, one-class SVM is, in analogy, an open-loop system. The criteria setting have to initiate from the primary problem for one-class SVM since there is no feedback information. The proposed automated model selction method based on genetic algorithms is by iteratively optimizing the primary objective function. The method is validated through constructive experiments.Invariance kernels / local kernels, such as Gaussian kernel and Laplacian kernel, can cause local risks and also the basic merics are short of flexibilities. To deal with such problems, a global kernel is constructed to improve the local kernels. The gains are duals: 1) The global kernel adds the global elements for local kernels; 2) The L1-distance in Laplacian kernel and L2-distance in Gaussian kernel, affected indirectly by the global kernel, obtain their respective flexibilities for dealing with the dependencies between samples. The performance improvements are validated by experiments.In support vector classification, Gaussian kernel has one globally optimal width for a specific task. However, Gaussian kernel is not well adaptive to the uneveniness of pattern space. There exist underfitting learning for sparse areas and overfitting learning for dense areas. Two local improvement methods are proposed for Gaussian kernels. One method is by a pseudo-conformal transformation to add the flexibilities for Riemannian metric induced on Riemannian manifold in reproducing kernel Hilbert space. That is, a large metric is used in sparse areas and a small metric in dense areas. The other method is to use a large kernel width in sparse areas and a small kernel width in dense areas. Though some local improvements are obtained for Gaussian kernel, there exist spaces in algorithms keeping to be further improved.In support vector classification, the importance of each dimension in feature space is depicted clearly by weight vector. However, Gaussian kernel doesn't differentiate the importances of dimensions (attributes) in input space. Obviously, the contributions of different attributes are also different, and to emphasize this, the Gaussian kernel with multiple widths is proposed. The new kernel increases the paramters for SVM and correspondingly, the multiple paramters's model selection algorithms are also proposed. All the ideas are tested by experiments.