The Study On The Sparse Structured LSTSVR Algorithms

Twin Support Vector Machine(TSVM)is a new machine learning method based on Support Vector Machine(SVM).It has good learning performance and has become a research hotspot in the field of machine learning.TSVM is often used to solve classification and regression problems.For classification problems,the twin support vector classifier(TSVC)aims to find a pair of non-parallel classification hyperplanes;for regression problems,the purpose of TSVR is to generate a pair of non-parallel regression hyperplanes on both sides of training sample points to determine the insensitive upper and lower bound functions of regression functions respectively.In order to simplify the calculation of TSVR,the least squares twin support vector regression(LSTSVR)simplifies the quadratic programming problem of TSVR into two linear equations by introducing least squares loss,thus greatly reducing the training time.However,LSTSVR minimizes the empirical risk based on least squares loss,which will leads to the following shortcomings:(1)the problem of “over-learning”;(2)the solution of model lacks sparsity and it is difficult to train large-scale data.This paper makes a the oretical analysis of LSTSVR's over-learning and lack of sparsity in the training stage,and studies how to solve the two problems of LSTSVR.For the problem(1),based on the principle of structural risk minimization,this paper adds a regularization term to the LSTSVR to control the complexity of the model,and proposes a structured least squares twin support vector regression(S-LSTSVR)to improve the generalization ability of the model.For the problem(2),Firstly,the dual model S-LSTSVR of S-LSTSVR is deduced,and the reason why the model is not sparse is analyzed.Secondly,the original S-LSTSVR(P S-LSTSVR)is given by using the representation theorem,which further proves the equivalence of PS-LSTSVR and dual S-LSTSVR in the regression problem.Finally,the sparse algorithm(SS-LSTSVR)for solving PS-LSTSVR is given based on incomplete Choesky decomposition,which is extended to the application of large-scale data.In this paper,the generalization ability of the S-LSTSVR model is proved by experimental analysis on artificial data.The sparsity of SS-LSTSVR algorithm is verified by experiments on four medium-scale datasets in UCI.Furthermore,the SS-LSTSVR algorithm is extended to large-scale data sets,and two large-scale data sets are used to verify the ability of SS-LSTSVR algorithm to deal with large-scale data sets.