Optimal Kernel Methods

In the early of 1960s, the theory of machine learning based on the data has begun to be researched. It has made great progress after 40 years research. In the nineties of the last decade, Vapnik and his group completed the statistical learning theory and constructed a general and effective machine learning algorithm—Support Vector Machine (SVM). SVM combines the advantages of both statistical learning theory and kernel method, and effectively controls the capacity of the hypothesis function set, which directly result into a good generalization performance. Many novel kernel machines appear in this period such as Kernel Fisher Discrimination (KFD), Kernel Principle Component Analysis (PCA), Relevant Vector Machine (RVM), etc, which is mainly enlightened by the successful applications of kernel functions in SVM. In the nowadays, researchers proposed the other successful learning machine, Kernel Matching Pursuit (KMP), based on the kernel technology. KMP could almost reach the equivalent performance compared with the classical SVM, while has the sparser solution. At present, all of these learning machines have been successfully used in pattern recognition, regression estimation, function approximation, density estimation, etc. From the viewpoints of the optimal kernel method, this dissertation includes five parts work as follows:1. We proposed a method to pre-extract support vectors before the training of support vector machine. As all we know, training a support vector machine (SVM) will cost quite more time, which is equivalent to solving a linearly constrained quadratic programming (QP) problem in a number of variables equal to the number of data points. This optimization problem will be challenging when the number of data points exceeds few thousands. Also, it is well known that the ratio of support vectors (SVs) is far low in many practical circumstances, and the decision made by SVM is only relate to these support vectors and has nothing with the other data. So the method of how to pre-extracting SVs becomes a novel task in SVM fields. In this dissertation, we introduce a new method for pre-extracting SVs based on vector projection and the geometrical characteristic of SVs, which could reduces the training samples greatly and speeds up the SVM learning, while remains the generalization performance of the SVM.2. For the improvement the sparsity of the SVM, an adaptive simplification strategy is proposed to simplify the solution of the SVM. In usually, SVM is currently considerably slower in test phase caused by numbers of the support vectors, which has been a serious limitation for the real time applications. So how to simplify the solutions of SVM becomes a key problem when using SVM in the online task. To overcome this problem, we proposed an adaptive algorithm named feature vectors selection (FVS) to select the feature vectors from the support vector solutions, which is based on the vector correlation principle and greedy algorithm. Moreover, the selection of number of the feature vectors can be controlled directly by the requirements, so the generalization and complexity trade-off can be controlled adaptively.3. Two efficient Mercer permitted kernels are constructed in this dissertation. The common used kernel functions in the kernel learning machines are not the orthogonal basis in the Hilbert space, which will directly degrade the learning machines' accuracy. Wavelet technique has been successfully used in the signal processing for its excellent approximation performance, which is also a complete orthogonal basis in the Hilbert space. In order to make use of the advantages of the wavelet, we have constructed the wavelet Mercer kernel and MRA Mercer kernel. Combining the machines with these two kernels can make up the defects of the traditional kernel functions and improve the performance of the learning machine.4. A new kind of machine named fuzzy kernel matching pursuit is proposed. In the practical, the recognitions of following problems are much important. One is to identify the special target, such as the detection of the cancer, aggressive plane, etc. The other is to recognize the minor target from the large data. Yet the conventional KMP machine treats all training data without difference, and gives the minimal errors of the whole dataset. Therefore, it doesn't present high performance to the signified patterns. In order to solve such problems, we proposed an effective machine, fuzzy kernel matching pursuit (FKMP), which imposes a fuzzy factor on each training data such that different points can make different contributions to the learning decision. The fuzzy factor can be pre-chosen according to the task's request. As the result, the detections of these special patterns are well solved.5. In the solution of the large scale problems, we need to resolve it quickly and hope to obtain the high performance simultaneous. The current computers are strong enough to deal with the complex science computations, while they still will be invalid when the scale of the problem exceeds some limit. Otherwise, processing large scale problem is an important problem in the real engineering application. In order to deal with such problems, we propose to combine the ensemble strategy with the machines and constructed the kernel matching pursuit classifier ensemble. The ensemble system separates the original images into several small sub-problems firstly, and then processes these individual problems one by one. Finally, it aggregates all the predictions and acquires the synthetic prediction. The greatest advantage of such strategy is that the system does not lose any information included in the original data and greatly improve the recognition performance simultaneous.