Research And Application Of Robust Least Squares Support Vector Machines

Binary classification is a wildly studied topic in statistical learning theory, machine learning and artificial intelligence. Support Vector Machines (SVM) adopts structural risk minimization principle and kernel method. It is a simple quadratic programming and has a unique solution. The objective of the Least Squares Support Vector Machines (LS-SVM) is a sum of squares error (SSE) term, thus its solution is obtained by solving a linear formulation equations, which makes it easier to be solved. The drawback of LS-SVM is that sparseness is lost in the solution because the use of equality constraints and the SSE. The solution in LS-SVM is also less robust.The real data sets are often accompanied with noise and uncertainty because of the randomness and no randomness. The noise and uncertainty may have a great impact on the classification model, which reduce the classification accuracy and the generalization ability of the model. Both SVM and LS-SVM adopt fixed objective function, which is a statistical learning method based on prior knowledge. The model construction in SVM and LS-SVM may not be adaptive to various kinds of data sets, thus makes the generalization worse. This thesis is mainly focus on how to improve the sparseness and robustness of LS-SVM and increasing its generalization ability.1. This thesis made a systematic review on how to improve the robustness of SVM and LS-SVM. We also pointed out the drawbacks of the existing models, from which we derived our main research topics, i.e. how to obtain an efficient binary classification model based on previous LS-SVM and how to improve the sparseness, robustness and interpretability of the model.2. Concentrating on the less robustness and sparseness of LS-SVM, We proposed to use the kernel principle component analysis (KPCA) to reduce the noisy features of the data sets. Based on the original work on how to increase the sparseness of the LS-SVM, we gave a bi-level L1 LS-SVM model-KPCA-L1-LS-SVM. KPCA can efficiently extract features from the original features and the usage of L1 in the objective function of the programming makes KPCA-L1-LS-SVM efficiently reduce the effect of the noisy data on the model, which reduces the computational complexity. Several tests on the simulation and benchmarking data sets prove the efficiency of KPCA-L1-LS-SVM.3. The existence of noisy data and features makes the labels of the sample data uncertain in binary classification. An efficient classification model can automatically determine which the relatively important data are and which are less important. The less important data play a lesser role in the construction of the separating hyper-plane. The idea of fuzzy membership can be used to describe the uncertainty of the labels. By adopting the fuzzy membership and the L1 norm in the objective function, we proposed a new model, which is called fuzzy-L1-LS-SVM. The numerical tests on this model proved that it can get rid of the impact of noisy data on the solution and had good interpretability.4. Different data plays a different role in the construction of the decision function. The more important the information contained in the data, the more important in the construction of the separating plane. To differentiate the different role of the data in the formulation of the decision function, the thesis proposed to assign a heavy weight on the more important data, while the less important data may be assigned a small weight. The weight can also get rid of the negative impact on the classification model to some extent, thus makes the model a robust one. The use of fixed Lp norm in the objective function in SVM and LS-SVM is not a data-driven model, which makes it less suitable for various complex data structure. In order to be more adaptive to the data structure, a weighted robust LS-SVM model is proposed. The simulation and the UCI benchmarking data tests proved that the model is robust, sparse and have good interpretability.5. The credit evaluation data sets have a very special data structure, which has unbalanced category. We tested the three models on the two UCI credit data sets and a credit data set of an anonymous American bank to prove the efficiency of these three models. The results showed the models are efficient in handling the kind of unbalanced data sets and can be an alternative tool in credit risk evaluation.