The Application Of Convex Optimization In Large Scale Machine Learning

Convex optimization is a subject which is having widly used in kinds of fields.It can obtain the optimal solutions from the solution spaces.The application of convex optimization in large scale machine learning is presented in this paper.This paper focused on a parallel learning algorithm for solving large scale kernel logistic regression(PDS) problem based on Fenchel Duality theorem and large margin nearest neighbor classifiers via cutting plane algorithm(LMNN_CPA).Compared with Support Vector Machine(SVM),there are two immediate advantages of KLR algorithm:(1)Besides giving a classification rule,the KLR also offers a natural estimate of the probability.(2)The KLR can naturally be generalized to the multi-class case through kernel multi-logit regression.Unlike SVMs,logistic regression does not yield sparse solutions,in the sense that all examples influence the solution,so the KLR algorithm for single machine can not deal with large scale datas.Learning the classifiers on subsets of training data run independently when block-update methods are employed.A simple customer-servicer parallel computing mode is designed that each customer node learns a sub-problem for the subset of training data. Service node receives the messages passed by all customer nodes after one optimization iteration is end,following by updating the objective functions of sub-problems.In comparison to non-parallel learning algorithms on standardized-datasets, we obtain encouraging results.Large margin nearest neighbor classifier using gradient optimization method is prone to local minima.In this paper, we also present a Mahalanobis metric learning method based on cutting plane algorithm which reduces largely constraints for solving the semidefinite programming problem.Experimental results on the UCI data sets show that our method can achieve promising speedups compared with the gradient based method under the similar training,test error rates.This paper has done the following work:(1) Proposed the theoretical analysis of the parallel solver for kernel logistic regression classification algorithm based on the Fenchel Duality theory and the solver for the subproblem. The main idea is translating large scale convex optimization problem into lots of small scale convex optimization,using the Fenchel Duality theory.(2) Given the software of the parallel solver for kernel logistic regression classification algorithm based on the Fenchel Duality theory. Firstly, giving the function chart and analyzing how many classes the software needs and how to design the classes. Then giving the flow chart of the server and the client.Finally, building the experiment platform,using the standard data sets from LIBSVM, doing lots of experiments.(3) Presented a Mahalanobis metric learning method based on cutting plane algorithm which reduces largely constraints for solving the semidefinite programming problem.Experimental results on the standard UCI data sets show that the method can achieve promising speedups compared with the gradient based method under the similar training, test error rates.