ADMM-type Algorithms For Regression Problems Based On Regularization

At present,regression problem is the research hotspot of machine learning,which is widely used in many fields such as life,management,transportation and so on.It is training specific regression model to learn the knowledge or experience of given training sample,and using the learned knowledge or experience to make a prediction for the decision-maker,so the process can help decision-maker better deal with the things that may happen in the future.The first chapter introduces the research background,research status and the main work of this paper.The second chapter introduces the alternating direction method of multipliers(ADMM).This chapter introduces two block ADMM algorithm and three block ADMM algorithm,and then this chapter introduces the convergence conditions of two block ADMM algorithm.In the third chapter,we use ADMM-type algorithm to get the optimal solution of logistic regresson regularization problem.Logistic regression is an important classification method in machine learning.The model is easy to fall into over-fitting when we are classifying massive data.Regularization is the main method to solve logistic regression fall into over-fitting.In this paper,we study the improved alternating method multiplier method(ADMM)algorithm to slove logistic regression models with different regularizations(L1 regularization,L2 regularization,L1-L2 regularization and Huber regularization).First of all,the regularization problem is divided into different blocks according to the structure of the problem.And then use a two-step ADMM algorithm to solve the problems:1)linearize sub-problems and obtain the approximate solution of the problem by using ADMM algorithm to train partial training samples;2)the approximate solution regard as the initial solution when apply ADMM to train all training samples.The numerical experiments show that the improved algorithm proposed in this paper is faster than the ADMM standard algorithm in some extent.At the same time,the numerical results also show that L1-L2 regularization and Huber regularization are superior to L1 regularization and L2 regularization in speed and quality in the comparison of regularization.In Chapter four,the regularization problem of robust linear regression is introduced and solved by ADMM algorithm.Linear regression is an important model in machine learning,but too large data will make the linear regression model fall into over-fitting problem.Regularization is the main method to solve the over-fitting problem.The standard regularization model is the method that linear regression co-operate with regularization to ensure the quality of classification and reduce the risk of falling into over-fitting.However,the classifier obtained by the standard regularization model will be unstable when the training samples are disturbed.To solve this problem,we propose two kinds of robust regularization models : 1)stochastic robust linear regression regularization: combining the expectation of residual with the regularization;2)worst case robust linear regression regularization: combining the worst case residual with the regularization.Then,we use ADMM algorithm to get the optimal solution of the robust regularization model.Numerical experiments show that the classifiers obtained by stochastic and worst-case robust linear regression regularization have good robustness when training disturbed data sets,but the classifiers obtained by standard regularization method fluctuate greatly.In the last chapter,we draw some conclusions and give some future research directions.