Study On The Key Problem Of The Competing Learning Vector Quantization And Support Vector Machine

The machine learning problem is an important aspect in modern intelligencetechnology because data are more and more important to our life. The main researchobjective of the machine learning is to find the rules that cannot be gotten analyticalfrom any sampled data. The rules are used to analyze the research objective and toforecast the future data or the data that cannot be observed. There are two kinds oflearning methods basically: supervised learning method and unsupervised learningmethod.The main researches in this dissertation include two parts also. One is on theunsupervised learning method, where the Learning Vector Quantization (LVQ),especially the Generalized Learning Vector Quantization (GLVQ), is studied. The mainresults are described in Chapter Two. Another is on some key problems of the SupportVector Machine (SVM), including the training algorithms of SVM and model selectionof SVM. The main results are introduced in Chapter Three to Chapter Eight. All of theresearch results can be discribled in the following six parts:1. The study on the unsupervised learning method aspect. Many Learning VectorQuantization methods are analyzed and some problems are pointed out. Thena new Revised Generalized Learning Vector Quantization (RGLVQ) isproposed. The new method not only is very simple but also overcomes somebugs of the current methods, such the "Scale"problem and sensitivity to theinitial points and initial learning rate. Furthermore, by introducing astimulating coefficient in competing step, a new competing technique toimprove the performance of the LVQ neural network is proposed also.2. The study on the standard Support Vector Machine. Three algorithms fortraining SVM are proposed. 1) For linear classification problem, the primequadratic programming is translated into an unconstraint nondifferentiableoptimal problem with lower dimension. A proper smoothing function is used todeal with the proposed optimal problem, and the Newton algorithm is selectedto solve the smoothing optimal problem. 2) Using the technology of theLagrangian dual, an unconstraint nondifferentiable optimal problem withlower dimension is proposed as the dual of the quadratic programming ofSVM. A strictly convex entropy function is used to deal with thenondifferentiable optimal problem, and then the perturbation solution of SVMis achieved by minimizing the strictly convex entropy function. 3) By themethod of the Lagrangian dual, an unconstraint and nondifferentiable optimalproblem in input space is proposed as the dual of the quadratic programmingof SVM. Using the Huber approximation function, the nondifferentiableproblem is approximated by minimizing a piecewise quadratic function. Andprove that minimal solution of the piecewise quadratic function iscorresponding the ε-support vector of SVM. The problem is solved by a fastNewton-type algorithm with exact line search. All those being done speedilyowes to the characteristic of the piecewise quadratic function whose gradient isa piecewise linear function, and the line search process is completedeffectively.3. The study on the reformation of SVM (1). An unconstraint differentiableconvex program is proposed as the Lagrangian dual of a simple SVM'sreformulation which is proposed by O. L. Mangasarian and his students. Theresulting problem minimizes a differentiable convex piecewise quadraticfunction in the input space. By the characteristic of the piecewise quadraticfunction, and combining the speedy exact linesearch method, a ConjugateGradients Support Vector Machine (CGSVM) is proposed to solve the problemquickly. The nonlinear classification problem is dealt with also by factorizingthe kernel matrix.4. The study on the reformation of SVM (2). A problem which minimizes aconvex piecewise quadratic function is also gotten as the Lagrangian dual ofMangasarian et al's reformulation. The semi-smooth method is introduced toshow that the unique minima solution of the resulting problem is justcorresponding to the solution of the system of the semi-smooth equations. Bythe characteristic of the piecewise quadratic function, an Exact SemismoothNewton Support Vector Machine (ESNSVM) is proposed. Because of thespeedy exact linesearch method, simple update of the Newton matrix and theQ-quadratic rate of convergence, the proposed method is very competitivewith the similar algorithms.5. The study on the geometric algorithm. Based on the geometric interpretationof SVM and some geometric algorithm such as nearest point algorithm, S-Kalgorithms et al, a new feasible direction algorithm is proposed. Twowell-chosen points are used to updating the current solution per iteration, butnot the same as S-K algorithms only one of them is used. The algorithm can beused to learning the separating SVM and the non-separating SVM with L2 lostfunction. Also from the plain geometric interpretation of v-SVM and thedefinition of the Reduced Convex Hull, the proposed algorithm is generalizedto train the v-SVM with commonly L1 lost function. Comparativecomputational evaluation of our algorithm against similar SVM methods suchas S-K algorithms and Generalized S-K algorithms shows that our algorithm isvery competitive.6. The study on kernel functions and model selection of SVM. The stretchingratio of the kernel function is defined in order to analysis the performance ofthe kernel function, and then a new form kernel function is introduced asreformation of Gaussian kernel function. The new kernel function has manyproperties as good as or better than Gaussian function, such as the new kernelfunction is always magnifying the distance between vectors in local area butwithout largening the radius of the circumscribed hypersphere enveloping allof the training samples. Especially, the experiment results show that thegeneralization errors gotten by the new kernel function is not sensitive to thetradeoff parameter C. Two criterions are provided to choose the betterparameter for a given kernel function. The first is the distance criterion whichminimizes the sum-square distance between the labeled training sample and itsown center, and maximizes the sum-square distance between the trainingsample and the other labeled-center. This is equivalent to the famous Fisherratio. The second is the angle criterion which minimizes the angle between thekernel matrix and targets matrix. The experiments show that our method isefficient.