Research On Robust Learning Models And Algorithms Of Support Vector Machine

Support vector machine (SVM) based on the statistical learning theory and structural risk minimiza-tion principle, has been an outstanding machine learning technique for classification and regression estimation problems. It has remarkable advantages such as the simple model structure, global optimal solution and good generalization performance. However, in practice, sampling errors, modeling errors and instrument errors may corrupt the training samples with outliers. The classical SVM yields poor generalization performance in the presence of outliers. From the perspective of loss functions, we make use of the optimization theory and methods, propose robust support vector machine models and algorithms, and make it more effective in practical problems. This dissertation is structured as follows:1. The classical support vector machine is sensitive to noises and outliers. We propose a general-ized linear non-convex loss function with flexible slope and margin to suppress the impact of outliers, and build a robust support vector regression. The robust model is more flexible in dealing with re-gression estimation problem. Meanwhile, it has strong ability of suppressing the impact of outliers. However, the generalized loss function is neither convex nor differentible. First, we approximate it by combining two differentiable Huber functions, and the corresponding optimization can be rewritten as a difference of convex functions (DC) program. Furthermore, we exploit a Newton algorithm to solve the proposed robust model. Meanwhile, we discuss the convergence property and computational complexity. The numerical experiments on benchmark data sets, financial time series data sets and document retrieval data set confirm the robustness and effectiveness of the proposed method.2. In the interest of deriving regressor that is robust to outliers, we set the upper bound of quadratic insensitive loss function as a constant, and propose a support vector regression (SVR) based on non-convex quadratic insensitive loss function with flexible coefficient and margin. The proposed loss function can be approximated by a difference of convex functions. The corresponding optimization can be formulated as a DC program. We utilize a Newton method to solve it. Numerical experiments on six benchmark data sets show that it has strong ability of suppressing the impact of outliers while keeps the sparseness.3. To overcome the limitation of LS-SVR that it is sensitive to outliers, we propose a robust least squares support vector regression (RLS-SVR), which employs non-convex least squares loss function. Non-convex loss function gives a constant penalty for any large outliers, and can be expressed by a difference of convex functions. The resultant optimization is a DC program. It can be implemented by utilizing the Concave-Convex Procedure (CCCP). RLS-SVR iteratively builds the regression function by solving a set of linear equations at one time. Numerical experiments on both artificial data sets and benchmark data sets confirm the promising results of the proposed algorithm.4. Least squares support vector machine is sensitive to large noises and outliers since it adopts the squared loss function, and results in bad robustness. To solve this problem, we employ Laplace loss function to weaken the effects of outliers and establish a robust least squares support vector regression model. The Laplace loss function is not differentiable, and the corresponding optimization problem is difficult to be implemented by classical convex optimization algorithm. We use a Huber loss function to approximate it and develop a Newton algorithm to solve the robust model. Numerical experiments on both artificial data sets and benchmark data sets demonstrate that compared with the classical least squares support vector machine, our proposed model can obtain better robustness.