Learning Theory Of Regularized Additive Models

With the rapid development of information technology,a large number of high-dimensional data have been accumulated in many application fields.It is very important that how to design a learning system to extract key variables of high-dimensional data and achieve effective prediction.As a natural extension of linear models,additive mod-els retain the interpretability of linear models and improve the representation flexibility of learning model in characterizing high-dimensional nonlinear function relationships.Recently,additive models have been extensively studied from aspects on model design,theory foundation,and application in the fields of statistics learning and approximation theory.Although these theoretical advances have deepened our understanding of addi-tive models,most of the current analysis is limited to the data-independent hypothesis spaces.The data dependent hypothesis spaces usually can provide much flexibility and adaptivity for algorithmic design and applications,which have been used successfully for the kernel-based regularized regression.Hence,it is necessary and important to explore the algorithm design and learning theory analysis for additive models under data-dependent hypothesis spaces.This is the main concern of this paper.Now,we summarized the main works as below:Firstly,we formulate a new additive regression model with coefficient-basedl_q-norm(1?q?2)regularization,and establish its error analysis and learning rate estimation.In order to obtain the upper bound estimation of excess generalization error,we decompose it into three parts:the sample error,the hypothesis error,and the approximation error.The sample error is bounded by the concentration estimation with the empirical covering numbers,and the hypothesis error is estimated by introducing the intermediate function.Finally,the approximation error is investigated by the kernel integral operator approximation.Theoretical results show that the proposed additive regression model can achieve fast convergence rate.Secondly,we formulate a new additive classification model with coefficient-basedl₁-norm regularization and establish its error analysis on the misclassification error.A new sparse additive classification model is constructed by extendingl₁-norm linear support vector machine to the case of kernel additive models.In theory,a new error decomposition is constructed based on the characteristics of learning model,and the hypothesis error is investigated by developing the estimation technique associated with hinge loss and reproducing kernel Hilbert space.At the same time,the uniform esti-mation of the sample error is obtained by the analysis technique based on the empirical covering numbers.Finally,the upper bound of the misclassification error is established for the proposed model.Thirdly,we formulate a new group sparse model with the coherence loss and establish its learning theory analysis on misclassification error and variable selection consistency.According to the theoretical characteristics of the coherence loss and group sparse regularization,the proposed approach can perform nonlinear classifica-tion,grouped variable selection,and class conditional probability estimation simulta-neously.In addition,the proposed model can be effectively solved by GMD method.In theory,we establish the decomposition for the excess misclassification error by con-structing two intermediate functions.The upper bound of the sample error is estimated by using the capacity-based concentration estimation technique.The stepping-stone technique is developed to obtain the hypothetical error estimate.Finally,theoretical characterizations for learning rates and variable selection consistency are provided.In applications,some data experiments verify the effectiveness and competitiveness of the proposed algorithm.Finally,we investigate the generalization ability of coefficient-based regularized regression with Fredholm kernel.Fredholm kernel is constructed by combining an outer kernel(associated with labeled and unlabeled data)and an inter Mercer kernel,which has shown excellent performance in density estimation and semi-supervised clas-sification.In essential,the hypothesis space is double data-dependent,induced by the data-dependent Fredholm kernel and its expansion with empirical observations.In this paper,Fredholm regression model is constructed under the framework of coefficient regularization,and the generalization error is established by combining the covering number and operator approximation techniques.The validity of the proposed approach is verified by simulation and real data experiments.In addition,error analysis for Fred-holm kernel ranking is provided.