The Learning Theory For Regularized Regression Model Under Several Learning Schemes

The regularized regression model has a good property to overcome the overfitting in machine learning,and it is a classic learning model with great generalization ability in learning theory.With the rapid development of data science today,learning schemes like ensemble learning,multi-task learning and deep learning are playing more and more important roles in many fields.The study of regularized regression model under these learning frameworks has also become increasingly important.We mainly study the properties of the regularized regression model in distributed learning,distribution regression and neural network in this article with six chapters.In Chapter 1,we describe the background and motivation of our study by stating the importance of learning theory.We mainly introduce the main tasks of machine learning and the learning schemes discussed in this article.Besides,we give the summary of this article in Chapter 1.In Chapter 2,we introduce the regularized regression model based on least squares loss,and we analyse the learning rate of HK norm regularized regression by using the classical probability inequality.And by utilizing the tricks of integral operator and the approximation of hypothesis space,we state the learning rate of l~2regularized regression and l~1regularized regression respectively.We also introduce the support vector machine which based on the regression model with hinge loss,and by doing so,we show that classification is a special case of regression.In Chapter 3,we study the learning theory of regularized regression model under distributed learning scheme.In distributed learning,we divide the entire sample set into multiple subsets,and train a local model on each subset of samples.Our final training result is obtained by taking the weighted average of all local models.We mainly study the property and the generalization of coefficient regularization under distributed learning scheme based on the study of Yuchen Zhang.By using the properties of RKHS and the tricks of integral operator,we prove that such regularized regression model can still get the optimal learning rate under this learning scheme.Compared with the spectral algorithm given by Zhengchu Guo,our result releases the constrain of hypothesis space.And our learning scheme also concurs the saturation effect in the study of Yuchen Zhang by improving the learning rate when the regression function has a stronger regularity.Meanwhile,we introduce the distributed learning algorithm with communication which is our study direction in distributed learning now.Under this algorithm scheme,local machines share information with each other meanwhile maintaining the privacy.We give the learning rate of this algorithm scheme by analyzing the iterative algorithm.In Chapter 4,we study the distribution regression problem.As the mathemati-cal foundation of multi-instance learning,distribution regression focuses on regressing from probability measures to real-valued responses.On one hand,we study the stability of distribution regression with dependent sampling,by utilizing the properties of strong mixing sequence and the covariance inequality based on strong mixing sequence,we prove that distribution regression still reach the optimal learning rate when the sampling sequence satisfies the strong mixing condition.Thus we extend the theoretic result given by Zhiying Fang to dependent sampling case.On the other hand,we give the learning rate of distribution regression with l~2coefficient regularization by using the properties of RKHS and the tricks of integral operator like those in chapter 3.Our learning scheme conquers the saturation effect occurring in the study of Zoltan Szabo.In Chapter 5,we state our main study direction by introducing the compression of neural network.We introduce the study about the compression of neural network given by Taiji Suziki.We get the approximation rate of the discrete neural network by using coefficient regularization scheme,and get the generalization ability of discrete neural network by the complexity analysis given by Schmidt-Hieber.In this chapter we also introduce some of our ideas and expectations in the compression of neural network.In Chapter 6,we give the summary of our article.