Reserarch On Support Vector Machine Incremental Learning

Most machine learning algorithms treat the training session as a static process: the algorithms use a large number of samples to train learning models,then the trained models are used to accomplish the tasks of prediction,classification,regression and so on.However,in many situations,new data is usually arriving sequentially,which means that the new knowledge could emerge among the new data,thus,the current trained model should be updated frequently.The learning of human is also a procedure of incremental updating with the new knowledge.However,the static machine learning process cannot meet the need of continuous learning.Therefore,recently great attention among the research communities has been focused on creating new learning algorithms that can meet the incremental learning needs.Support vector machine(SVM)is an important machine learning method based on statistical learning theory.A support vector machine constructs a hyperplane or a set of hyperplanes through solving the convex programming problem.In the scenario of data arriving sequentially,we usually face the challenges of insufficient initial training samples and the learning model should be updated since the new knowledge is appearing.In this thesis,we propose methods based on incremental SVM to address such challenges.The main works are focused on improving the basic theory,the procedure,the efficiency and the performance of incremental SVM.The contributions of this thesis are summarized below:1.According to the classic Karush-Kuhn-Tucker(KKT)theorem,at every step of incremental support vector machine(SVM)learning,the new adding sample which violates the KKT conditions will be a new support vector and migrate the old samples between support vector set and non-support vector set,and at the same time the learning model should be updated based on the support vectors(SV).More SVs will result in better accuracy but a slower speed.Additionally,the learning model will be frequently updated which is not always necessary and will not greatly increase its accuracy but decrease its efficiency.Therefore,how to choose the new SVs from old sets during the incremental stages and when to process incremental steps will greatly influence the efficiency and accuracy of incremental SVM learning.In this work,a new algorithm is proposed to select candidate support vectors and use the wrongly predicted sample to trigger the incremental processing simultaneously.Experiment results show that the proposed algorithm can achieve good performance with higher efficiency,faster speed and better accuracy than traditional methods.2.When started with very few samples,the initial model is not well trained and will be frequently updated during the later learning process.Since there's a little priori knowledge and the data distribution is unknown,if we directly use the traditional method to select all the violating KKT conditions samples to update the current classifier,the classification accuracy may greatly decline.In this work,a new strategy is proposed to select the candidate SVs by using the most important and the most informative criterions.Particularly,we apply a local-global regularization method during the online learning process to promote the speed and efficiency of the learning.Experiment results show that the proposed algorithm can achieve good performance with faster speed and better accuracy than traditional methods.3.The online Passive-Aggressive(PA)algorithm modifies the current classifier to correctly classify the current sample by updating the weight vector and remain the new classifier as close as possible to the current classifier.As a result,the efficiency of PA is low.The classic incremental SVM changes current model greatly for it updates current model whenever a new sample is violating the KKT conditions.However,the computational efficiency is very high because of the kernel computation.Both of the two methods will cost enormous time to get the optimal model especially when the training data is very large.In this work,we propose a new method that incorporates the PA to the classic ISVM algorithm.Experiments show that the proposed method has fewer update times than PA and better accuracy results than ISVM.The proposed method has a more aggressive characteristic than the PA algorithm and a more passive characteristic than the classic ISVM.4.When classifying very large-scale data sets,there are two major challenges: the first challenge is that it is time-consuming and laborious to label sufficient amount of training samples;the second challenge is that it is difficult to train a model in a time-efficient and high-accuracy manner.This is due to the fact that to create a high-accuracy model,normally it is required to generate a large amount of representative training data.A large training set may also require significantly more training time.There is a trade-off between the speed and accuracy when performing classification training,especially for large-scale data sets.To address this problem,a novel strategy of large-scale data classification is proposed by combining K-means clustering technology,outlier detection and multi-kernel support vector machine method.The evaluation results show that the proposed instance selection method significantly reduces the size of training data sets as well as the training time;in the meanwhile,it maintains a relatively good accuracy performance.Finally,the conclusions are made and the future work is presented.