Study On Least Squares Support Vector Machine And Its Applications

Statistical learning theory is a special statistical theory for small samples. It has built a theoretical structure for studying modern identification and machine learning problem on the finite samples. Support vector machine (SVM) which is developed on the basis of statistical learning theory is becoming a top topic on machine learning field for its theoretical foundation and generalization ability. SVM transforms the problem of searching for the optimal hyperplane between two classes into the problem of solving the maximal classification margin. The maximal margin problem is actually a quadratic programming (QP) problem subjected to the inequality constraints. Some researchers have proposed many efficient approaches to solve this QP problem. But the existing methods are time consumption and huge space requirement for large samples. Suykens proposed the least squares support vector machine (LS-SVM) which converts the inequality constraints into equality constraints. And the solving of SVM is transformed from a QP problem to a group of linear equations. LS-SVM simplifies the learning process of SVM, lowers the difficulties on solving and expands the application fields.In this dissertation, the followings are studied on the basis of LS-SVM:(1) Analyse the relationship between the LS-SVM for classification and regression, and prove the mutual conversion between them. Moreover, expand the binary-classification to multi-classification and propose a one-step method based on LS-SVM for regression. This method only needs to solve a group of linear equations. For the existing method on multi-classification problem, the classifier is usually constructed by decomposing and reconstructing procedure. The label of a sample is decided by several classifiers through voting scheme. We verify the mutual conversation between regression and classification through numerical simulation. Firstly, the function regression problem is solved by classification method, and the mean square error is satisfactory. So the function regression problem could be converted into classification problem. Secondly, the multi-classification problem is solved by function regression. We test on artificial dataset and benchmark dataset, and compare the accuracy of classification with that obtained on other multi-classification methods. The results show that the accuracy of the proposed method is comparable with other methods. So the one-step classification method based on LS-SVM for regression is efficient for multi-classification. The classification problem could be solved by function regression method.(2) Propose a set of processing method based on LS-SVM to deal with the keywords extraction problem for scientific literature. We process all documents with indicated keywords, collect all the given keywords and populate the keywords database with their unique set. The samples are designed as a 5 dimensions vector: the times that a candidate keyword appears in title, abstract, body and conclusion of a scientific literature document, respectively, and if it is the indicated keyword in the file. In order to construct the samples, we scan each line in the plain-text file for each item in the keyword database. There are two kinds of samples in the sample set. The one is keyword sample, and the other is non-keyword sample. The number of keyword samples is much smaller than that of the non-keyword samples. In order to balance the two kinds of samples we copy the small samples several times. The classifier is trained by LS-SVM for regression. We test the classifier on both files with indicated keyword and without indicated keyword. The results show that the proposed method is efficient for keyword extraction in scientific literature.(3) Present an iterative LS-SVR algorithm based on quadratic Renyi entropy. All the training samples are support vectors in LS-SVM. So, LS-SVM loses the sparseness of support vector which is one of the important advantages of conventional SVM. The proposed algorithm overcomes this drawback. The quadratic Renyi entropy is the evaluating criterion for working set selection, and the size of working set is determined at the process of iteration adaptively. For the entropy is a measure of system randomization, the big entropy means that the system is randomized well. The working set with big entropy could reflect the law of the sample set. LS-SVM constructed by this working set has better generalization. The regression parameters are calculated by incremental learning and the calculation of inversing a large scale matrix is avoided. So the running speed is improved. We experimented on several datasets. The training time spent on the proposed algorithm is 0.59%--15.4% of standard LS-SVM, and the number of support vector is 0.85%--6.54% of training sample for the similar regression accuracy and small mean square error. This algorithm reserves well the sparseness of support vector and improves the learning speed.(4) Put forward two gene selection algorithms based on LS-SVM. Both of them adopt sequential forward selection search scheme and the total number of genes processed by these proposed algorithms is smaller than that processed by other algorithms using support vector machines. The number of selected genes can be determined adaptively in both of the two proposed algorithms. The advent of DNA microarray technology has brought a widely use of gene expression for identification and classification in biology and medicine. DNA microarrays could provide useful information for cancer classification at the gene expression level. However, the number of genes in a microarray is always several thousands and the number of training samples always several dozens. Overfitting is a major problem due to the high dimension and the small data size. And because of so many genes in the microarray, it will be a big challenge on computation time and memory space. It is necessary to find a gene selection algorithm to deal with DNA microarrys. For the first algorithm, the change of objective function value is the criterion to select the genes. The gene with the smallest ranking score ( wi )2 is eliminated at back trace procedure. The number of selected genes and the classification accuracy on testing samples are similar to the literatures that adopt sequential backward selection search scheme. For the second algorithm, the leave-one-out error (LOOE) and C bound are used to value the gene subset. The candidate genes are ranked on the change of objective function. The number of genes selected by this algorithm is almost the same as that selected by LGFS. The proposed algorithm achieves 0 generation error but it does not imply that the classifier can classify any test samples with 100% accuracy. However, it is true that the lower generation error does indicate the higher generalization ability.LS-SVM will be used widely for its simplicity and generalization ability. In this dissertation we study the improvement and application of LS-SVM. This research work will promote the theoretical study of the algorithm and expand its application in the field of classification and function regression.