Application Research On Lp-Regularization In Support Vector Machines

The regularization method, which is an important mathematical approach in engineering The regularization method, which is an important mathematical approach in engineering and technology, can solve the original ill-posed problem with an approximate solution, while the performances of a machine-learning way with different regularization methods are diverse, so it is a top topic that how to effectively choose a regularization method in machine-learning area. The conventional statistical-learning way mostly commonly produces over-fitting issue, owing to just minimizing empirical crisis, and, to improve the performance of classifier, it is critical to add the amount of training samples and cautiously pick up the model of classifier with enough expertise experience. However, the statistical machine-learning way, coming up with intact and precise theories, such as VC dimension, generalization, the limit of generalization, structual crisis and so on, is capable of avoiding over-fitting issue, solving small-sample problem and enhancing the generalizaiotn of classifier, after simplifing the classifier which is trained by minimizing both empirical crisis and confidence interval. The perfomance of linear classifier would be poor if it classifies the non-linear samples, which can be overcome by non-linearly mapping the original samples space into features space with more dimensions or even infinite dimensions. In actual applications, the non-linearly mapping functions may be replaced by kernel functions provided that they satisfy Mercer theorem. Due to the over-fitting issue caused by only complying with ERM criterion in former machine-learning way, current Support Vector Machines are competent not only in solving it but also in solving small-sample problem and enchancing the generalization of classifier, through introducing Lp (p=2, p=1) regularization to reduce the VC dimension of classifier with constant empirical crisis abiding by SRM criterion. Our contributions are as follows:(1) The functional theories relavant to Support Vector Machines have been approached in detail, such as Direct Problem, Inverse Problem, Ill-posed Problem, Moore-Penrose Generalized Solution, Unstability of the Generalized Solution, Regularization Principle, Regularization methods, Statistical Machine-Learning Principle, VC dimension, Generalization, the Limit of Generalization, SRM Criterion and so on.(2) Linear Support Vector Machines, Kernel Support Vector Machines and Manifold Regularization Support Vector Machines with Lp (p=2, p=1) regularization have been approached in supervised learning framework and semi-supervised learning framework. The results of Compared experiments on two synthesic circles datasets and four real two-class datasets with those methods verify that Kernel Support Vector Machines linearly classify those non-linear samples effectively and introducing intrinsic geometric information of training samples may improve the performance of Support Vector Machines. In addition, the characteristics of datasets and eleven methods, as well as the ROC curves derived from those methods have been discussed.(3) Combining the sparseness and the merits of smooth functions, Smoothed Sparse Regularization Support Vector Machines has been proposed and its’ optimization process has been emphasized. With Gaussian smooth functions in approximating Lp (p=0) regularizer, a simpler classifier can be trained and the important features of samples will be selected. The results of experiments on five two-class datasets with this method and current linear Support Vector Machines, verify its effectiveness. Meanwhile, it is discovered that the instrinsic classifier of multi-class dataset may be complex and the performance of classifier trained may not be better with simpler model.