Consistent Statistical Inferences For High-dimensional Linear Models And Partially Linear Models

In modeling procedure a large number of predictors usually are introduced at initial stage as a full model. But when it is applied to real problems, lots of pre-dictors and large model will need heavy computation load and resource occupation, which may increase variance and mean squared error of the estimator, and even make the numerical algorithm unstable or abort abnormally. This will directly influence estimation of parameter and model prediction. In order to simplify the model and enhance predictability, some less significant variables are removed. So a reduced model (or restricted model) is formed. Under certain regularity conditions, the estimation of the remaining parameters in the reduced model is consistent. However, when the consistency of parameter estimation is specially taken into account for modeling with variable selection, the sparsity and oracle property (e.g. SCAD estimator) hold only in the pointwise sense, which means that such an estimator does not have a good glob-al property. Furthermore, for the case of restricted-model (RM) estimation or post-model-selection estimation, if some significant variables are unfortunately removed, even if their coefficients are nearly zero, the restricted model (or submodel) is misspec-ified and the popularly used estimators for remaining parameters converge to pseudo parameters rather than the true ones. Even if the selected submodel is only locally misspecified, the commonly used estimators for the parameters in the submodel are still inconsistent.On the other hand, in some application cases only part of the variables are inter-ested, because they can be easily controlled. While for the other variables, maybe they are not emphasized or cannot be properly controlled, or their influences on the response variable are not clear. If the full model is applied, the similar problems will be faced and the model may be misspecified. The submodel is highly biased if only regressed on the variables of interest, and so is the corresponding estimator. It is much reasonable to form a semiparametric model in which the variables of interest are parametric part and the other variables are nonparametric part, but the nonparametric part will be faced with the curse of dimensionality. Especially, when the dimension of the other variables are high the curse of dimensionality will ruin the estimation and prediction. In application fields more and more semiparametric models are to be faced.All the above indicate that, it is still a problem to consistently estimate parameters and construct confidence regions in the modeling process with variable selection or some variables specified. In this thesis, we shall mainly study some estimation methods for parameter of interest and the corresponding model prediction in the above cases. For linear regression model suppose that parameter β and covariate X are only interested. In chapter two, a semiparametric method is firstly proposed to adjust the biased submodel By finding a proper direction τ∈(?), a partially unbiased adjusted model is constructed, where g(τTZ)=E(Y-βTtX|τTZ)=-γTE(Z|τTZ). Such an adjusted model is partially locally unbiased in the sense that Applying univariate nonparametric kernel method to estimate the nonparametric term g(τTZ) as following in which K(·) is a kernel function, h is bandwidth depending on the sample size n. Substituting g(rTZ) into the adjusted model, an estimator of β is obtained as where It is proved that the estimator βA is (?)-consistent on the subset W. Furthermore, based on the idea of PT estimation with F-test, another estimator βAPT which is also relied on the full model is obtained. In the third section, a confidence region estimation is constructed by empirical likelihood method. Upon the method proposed in chapter two, the curse of dimensionality is avoided by a univariate nonparametric function. Furthermore, the property of inference is insensible to the direction τ, so the new method is robust. No matter how large the bias of the submodel, simulation results show that parameter estimation and confidence region of the new method are better than those of existing methods.Although the adjusted method proposed in chapter two can markedly reduce the estimation bias of the submodel, it only holds on a subset W of the covariates’support region. So in chapter three, based on the submodel Y=βT X+η and the parameter of interest β, a globally unbiased working model is constructed with E(Z)=0. The main idea is to firstly decompose covariate Z into independent components Z(1),…,Z(q)then make use of the independent components correlated with covariate X. For each component Z(l),a univariate nonparametric term gl(Z(l))=E(Y-βTX|Z(l))=γTE(Z|Z(l)), is added into the submodel to reduce the bias of the submodel.(1) When the covariate Z is normally distributed, the principal component regression (PCR) is applied, gl(Z(l))=αlZ(l) and the adjusted model is really a linear model where Z(l) is the l-th principal component of Z;(2) Otherwise, we adopt independent component analysis (ICA) method and Z(l) is the l-th independent component of Z. Based on this adjusted model, applying univariate nonparametric kernel method to estimate each nonparametric term gl(Z(l)) as following in which K(·) is a kernel function, hl is bandwidth depending on the sample size n. Substituting gl(Z(l)) into the adjusted model, an estimator of β is obtained as where It is proved that the estimator βA is globally consistent on the whole space of covariates X and Z, and it is also asymptotically normal. Because the added nonparametric parts gl(Z(l)) are independent of each other, large computation load is avoided which is faced in the backfitting method for general additive model. Furthermore, the computation error is fairly small if the number K of adjusted parts is not large. When Z is normally distributed, the corresponding result can be obtained by applying least squares method to the linear adjusted model.When the adjusted steps K is large or even close to the original dimension q of covariate Z, the method proposed in chapter three will result in large computation error and will lose its superiority. In chapter four, more generally, a two-stage remodeling and estimation method is constructed for sparse partially linear model in which parameter β is interested and parameter7is sparse. For simplicity, we assume that U is univariate and E(Z)=0. In fact,f(·) can be extended to additive strueture with multidimensional U.In details, in the first stage, making use of the independent components Z(j) correlated with covariate (X,U) and by a multi-step adjustment as that in chapter three, a globally unbiased model is reconstructed. In the second stage, we further reduce the adjusted model by a semi-parametric variable selection method proposed by Zhao and Xuc(2009). In details, by a truncated expansion of orthogonal series method, each nonparametric term gj(Z(j)) and nonparametric term f(U) are approximated with Then we estimate parameters β,θj and v with group SCAD method as following where Pλ(·) is the SCAD penalty function defined as satisfying a>2, ω>0, pλ(0)=0. Let Mn={1≤j≤Ko:θj≠0}, Kn=|Mn|. For simplicity, we suppose that Mn={1,2,…, Kn}. Denote gj(Z(j))=E(γTZ/Z(j)), j=1,???, Kn,ζKn=Y-βTX-g1(Z(1))-···-gKn(Z(Kn))-f(U), finally, we get model After two-stage remodeling the final model is sufficiently simplified and globally con-ditionally unbiased. In the theoretical results, the convergence rate of parametric estimator β and nonparametric estimators gl and f are proved, and the estimator β is proved to be asymptotically normal. Because variable selection much relies on the sparsity of the parameter, if we directly consider the partially linear model, some irrel-evant variables with X but with nonzero coefficients may be selected into model. This may affect the estimation of the parameter β on its efficiency and stability.In chapter five, for high dimensional linear model Y—βTX+γTZ+ε with normal covariates and error term, we combine the tilted variable method proposed by Cho and Fryzlewicz(2012) with the relaxed projection method proposed by Zhang and Zhang(2012) to remodel the biased submodel Y=βTX+η with parameter ft of interest. If γTE{Z|X)≠0, then E(η|X=x) is a nonzero function. So we shall firstly adjust the biased submodel with the method of Cho and Fryzlewicz(2012). After the components of Z correlated with X, which are denoted as ZCX, are added into the submodel, an adjusted model is obtained, where ζ=ε+∑k∈j\CX γkZ(k) and j={1,2,???, q}. Then we compute the tilted variables corresponding to design matrix X as where Πzx is the projection onto space spanned by ZCX.(1) If all the lengths of the tilted variables are not too short, based on the tilted variables Uo and the above adjusted model, parameter β can be directly estimated as It is proved that βT is consistent under mild conditions.(2) Otherwise, it need to relax the projection by the method of Zhang and Zhang(2012). In details, the tilted variables with relaxed projection are defined as where d-|CX|, tr(V) is the trace of matrix V, A is the penalty parameter,θ satisfying Applying the tilted variables U and based on the adjusted model, we can get a linear estimator Because the projection is relaxed, so it need to reduce the bias of βL. Suppose that (β(init),γ(init)) is an initial estimator of parameter (β,γ) of the full model, satisfying a new unbiased estimator of parameter β is constructed as following Finally, based on this point estimator, confidence region of β can also be estimated. It is proved that such a point estimator ftu of parameter β is consistent and coverage rate of the confidence region is guaranteed.