Parameter Selection For The Gaussian Kernel And Construction Of Orthogonal Polynomial Kernels

With the increment of data acquisition and storage capacity,the rapid development of computer operation ability,and the successful exploitation of hidden patterns or rules in the data by use of computers,pattern recognition has become an increasingly active research field.Kernel-based learning methods,namely kernel methods,provide high performance in many small-to medium-sized data community for pattern recognition in terms of rigorous mathematical foundation,classical modularity and efficient nonlinear operation.And they can effectively avoid some dilemmas such as “dimension disaster”,“local minima” and “overfitting”.The success of kernel methods is heavily dependent on the selection of the right kernel function and proper setting of its parameters.However,the choice of kernel function and its parameters is a challenging task.Therefore,kernel selection is of great theoretical value and practical significance for the good performance of kernel methods.Many successful kernels have been implemented,and we focus on the Gaussian kernel and the orthogonal polynomial kernels here.Based on its superior performance,the Gaussian kernel is widely used in various applications.Many universal kernel selection methods have been derived for optimizing the Gaussian kernel.However,these methods may have some disadvantages,such as heavy computational complexity,the difficulty of algorithm implementation,and the requirement of the classes generated from underlying multivariate normal distributions.To overcome these problems has become a trend in the kernel selection for the Gaussian kernel.In recent years,orthogonal polynomial kernels are widely studied and applied in various learning applications because their parameters can be optimized easily.However,finding an optimal kernel for a given data set from these orthogonal polynomial kernels is a challenging task.There are two main clues about kernel selection in this thesis: one is proposing some new kernel tuning criteria for the Gaussian kernel,and the other is presenting some valuable construction and selection guidance for orthogonal polynomial kernels.The main research findings are summarized as follows:(1)We propose the centered kernel polarization criterion to optimize the Gaussian kernel.To give a better correlation between the alignment value and the performance of kernel methods,we proposed the centered kernel polarization.This criterion works by maximizing the alignment of the centered kernel matrix and the centered target matrix.Compared with the kernel polarization criterion,this new criterion can decrease the influence of the coordinate origin setting and uneven data on the alignment value.The approximate criterion function can be proved to have a determined global minimum point.Based on this result,the optimal parameter can be located easily.It can be shown that the proposed criterion is equivalent to the maximization of the distance between the class mean locations in the feature space.So the obtained conclusion provides the theoretical foundation for the maximization of the distance between class centers.Comparative experiments are conducted on 23 benchmark examples with three Gaussian kernel based learning methods and the results well demonstrate the effectiveness and efficiency of the centered kernel polarization criterion and its multiclass extension.(2)We propose the generalized kernel polarization criterion to optimize the Gaussian kernel.By taking the within-class local structure into account,the generalized kernel polarization criterion has been proposed to tune the parameter of the Gaussian kernel for classification tasks.The criterion does better in maximizing the class separability in the feature space,and it can be seen as an extension of the kernel polarization criterion,the centered kernel polarization criterion and the local kernel polarization criterion.Furthermore,the formulated criterion function can be proved to have a determined approximate global minimum point.Besides,the local kernel polarization criterion function can also be proved to have a determined approximate global minimum point.This valuable characteristic makes the optimal parameter easier to be found by many algorithms.And the results enrich the research on the Gaussian kernel optimization.The generalized kernel polarization criterion and its multiclass extension are evaluated on 19 data sets with support vector machine,and the experimental results show the effectiveness of the proposed criteria.(3)After comparing the generalization performance of the Gaussianmodified orthogonal polynomial kernels,we propose the triangularly modified orthogonal polynomial kernels.The properties of the Gaussianmodified orthogonal polynomial kernel functions are studied.For unnormalized data,we give some suggestions on the selection of the right Gaussian-modified orthogonal polynomial kernel and the proper setting of its weight function.To overcome the difficulty of the parameter tuning of the Gaussian-modified weight function,a new set of triangularly modified orthogonal polynomial kernels is proposed.The parameter of the new proposed kernel can be chosen easily.By experimental results we show that the triangularly modified orthogonal polynomial kernels which integrate the scale invariance of the triangular kernel can provide competitive classification results.(4)We give some suggestions on the construction and selection of the orthogonal polynomial kernels.Some orthogonal polynomial kernels,which are based on different basis orthogonal polynomial functions,different modified weight functions and different construction methods,are compared and contrasted.We review the properties and the construction methods of these orthogonal kernel functions and highlight the similarities and differences between them.In classification and regression scenarios,we perform experiments on 32 normalized benchmark data sets for better illustration and comparison of these kernel functions.We give an overall comparison of these orthogonal polynomial kernels by using statistical analysis methods,and discuss the possibility of these orthogonal polynomial kernels to be used as general kernel functions by comparing with4 common kernels.The obtained results provide theoretical basis and technical support for the use of orthogonal polynomial kernels.Kernel selection is a fundamental problem in the study of kernel methods.We propose some criteria to optimize the Gaussian kernel,and give some valuable advice on the construction and selection for orthogonal polynomial kernels.The study in this thesis enriches the theories and methods of kernel selection,and it is of great theoretical and practical significance to kernel selection which exists widely in the research and application of kernel methods.