Research On Learning Algorithms For Support Vector Machines Based On Optimization Theory

Support vector machine (SVM) is a new approach that can solve machine learning problem with optimization methods. In recent years, there has been a surge of interest in SVM. It has achieved a prodigious progress in the theory research and algorithm implement, thus has been an active research area in machine learning.SVM translates the machine learning problems into the optimization problems and applies the optimization theory to construct algorithms. The optimization theory is the important theory foundation of SVM. This dissertation mainly does researches on SVM with the optimization theory and the optimization methods. All of the research results can be described as follows.1. The study on least squares support vector machine (LSSVM). A preconditioning conjugate gradient method for LSSVM is proposed. LSSVM has to solve a large scale linear system of equation when the number of the training simples is large. Block matrix is applied to reduce the system of equations. In order to improve the rate of convergence and overcome instability of numerical value, a preconditioning conjugate gradient method is presented for solving the reduced system of linear equations. The training efficiency of LSSVM is improved greatly by the method.2. The study on smooth SVM. The objective function of the unconstrained SVM model is non-smooth and non-differentiable. So a lot of optimization algorithms can not be used to find the solution of the model. To overcome the difficulty, a novel smoothing method using Chen-Harker-Kanzow-Smale functions for SVM (CHKS-SVM) is proposed. The Newton-Armijo method is adopted to train the smooth CHKS-SVM. Using the proposed method, the optimal separating hyperplane is trained in batches, both the training time and memories needed in the training process are saved. So the novel method can efficiently handle large scale and high dimensional problems.3. Based on KKT complementary condition in optimization theory, two unconstrained non-differential optimization model for SVM and support vector regression (SVR) are proposed respectively. A smooth approximate method is given to deal with the proposed optimization problems. An adjustable entropy function method is given to train SVM. The proposed method can find an optimal solution with relatively small parameters, which avoids the numerical overflow in the traditional entropy function methods. The adjustable entropy function method can be used to train SVR analogously, which avoids the numerical overflow effectively. It is a new approach to solve SVM and SVR.4. The study on fuzzy SVM. A fuzzy SVM based on border vector extraction is presented, which overcomes the disadvantage that traditional SVM are so sensitive to noises or outliers in the training samples. Select possible support vectors for border vectors to train SVM, so as to reduce training samples and improve training speed. The fuzzy membership, which is defined according to the distance between the center of their spheres and border vectors and outliers respectively, both diminishes the effect of noises and outliers and improves the role of support vectors to design a classifier. The conception of fuzzy membership is introduced into LSSVM in order to overcome the disadvantage that LSSVM is much sensitive noises or outliers in the training samples. And then fuzzy LSSVM (FLSSVM) is proposed based on support vector domain description. The new defined fuzzy membership can reduce the effect of outliers. The constrained convex quadric programming problem can be translated into positive definite linear equation system. The fast Cholesky decomposition is applied to solve the linear equation system. The regression performance of FLSSVM is superior to that of SVM and LSSVM.5. The study on Semi-supervised SVM (S3VM). A piecewise function is used as a smooth function and smooth piecewise semi-supervised support vector machine (SPS3VM) is given. The approximation performance of the smooth piecewise function is better than that of the Gaussian approximation function. According to the non-convex character of SPS3VM, a converging linear particle swarm optimization is first used to train S3VM. Experimental results illustrate that our proposed algorithm improves ?TSVM in terms of classification accuracy.