Study Of Online Learning Algorithms With Applications

Traditional classification methods in machine learning community, particularly thewildly used support vector machines (SVMs), are generally designed in the batch mode.In other words, the whole training samples need to be obtained before learning and to bepeocessed at once for the decision model. However, in practice, sometimes the data aretoo huge to be processed at once. More importantly, with the new data coming, thewhole data should be retrained, which costs lots of time and space resources. Aiming atthese issues, online learning althorithms whose goal is to update the model on eachround without retraining, as well as their applications, are discussed in this thesis. Themain contributions of this thesis are as follows,For SVM-related methods, a novel incremental learning algorithm called onlineindependent Lagrangian support vector machine (OILSVM) is proposed. As opposed tothe newly proposed OLSVM that utilizes the KKT conditions as data selection strategy,the size of the solution obtained by OILSVM using a linear independence check isalways bounded, which implies bounded memory requirements, training and testingtime. Further, online incremental learning algorithm based on successive overrelaxationis proposed for Twin SVM, where the update of matrix after increment is solved basedon the information at the last time step, without repeatative computation. Compared tothe batch method, the proposed algorithm can save training time greatly at the price of anegligible loss in accuracy.For perceptorn-like algorithms, an online instance weighting scheme based onpairwise distance is presented to cope with sensitivity of their recognition performanceto noise points and outliers. This method can be implemented incrementally, which isbenifical for online learning mode. Then we introduce this scheme to passive-aggressive(PA) online algorithm and produce a weighted PA (WPA) algorithm, which not onlyimproves the recongtion performance and robustness of conventional PA greatly, butalso enjoys the time effiency of Perceptron-like algorithms.Additionally, the validity and potential value in engineering applications of theproposed algorithms are supported by the expetimental results on machine learningbenchmark datasets, image datasets, as well as radar emitter real datasets.