Multi-class Probabilistic Classification Vector Machines Algorithm And Its Applications

As one of the fundamental problems in machine learning,classification has re-ceived extensive attention.So far,researchers have put forward various models and al-gorithms for solving this problem.Among them,there is one collection of algorithms:Sparse kernel method,which is applicable to the dataset with feature vectors as its input,has gained a great success and extensive attention in recent decades.All of these algo-rithms are proposed for the binary cases and cannot be directly applied to the multi-class cases.Probabilistic Classification Vector Machines(PCVMs)is the one that combines the advantages of Support Vector Machines(SVMs)and Relevance Vector Machines(RVMs).However,it is still only applicable to binary cases.Based on the Bayesian framework,we generalize the binary PCVMs models to multi-class version,and pro-pose a multi-class Probabilistic Classification Vector Machine(mPCVMs)model.Two algorithms are proposed,a top-down algorithm named mPCVM1 and a bottom-up al-gorithm named mPCVM2.The main work of this thesis is as follows:(1)The mPCVMs model proposed in this thesis is a generalization of the PCVMs model.Based on the Bayesian framework,it requires fewer pre-fixed parameters than SVMs.The mPCVMs model uses a truncated Gaussian prior,making the model weights consistent with the corresponding labels,avoiding the appearance of untruthful base samples and reducing the sensitivity of models with regard to some kernel parameters.The weights of most multi-class models increase with the number of dataset categories,in contrast,the number of weights of mPCVMs models does not change with the number of categories.When the number of categories is large,mPCVMs has a small number of weights,which reduces the time and space consumption for an optimization algorithm,and improves the numerical accuracy of matrix inversion operations.(2)This thesis presents an optimization algorithm based on the Expectation Max-imization(EM)which named mPCVM1.It attains the optimal weights by maximizing the posterior distribution of model weights.In the optimization process,all the base samples are initially included.In the iterative process,redundant and unrelated base samples are gradually removed to achieve sparsity.Due to this pattern many-to-less,we call this algorithm a top-down algorithm.(3)Based on the maximization of type-? likelihood of the mPCVMs model,this thesis proposes an incremental optimization algorithm:mPCVM2.This algorithm ini-tially adds a single base sample.During the iterative process,a base sample can be added,deleted,or modified.Because of this one-to-many fashion,we call this algo-rithm a top-down algorithm.(4)Through theoretical analysis and a large number of experiments,this thesis validates the effectiveness of the two algorithms.Especially when the dataset has a large number of categories,two algorithms proposed in this thesis have outstanding performances.Through comparing two algorithms in various aspects,we deliver some suggestions on how to choose between the two algorithms in practical applications.