Regression Under Massive Data Estimation Algorithm Of Quantile

Quantile regression has been widely used for more than 40 years since it was formally proposed because it has stronger robustness than mean regression and can more accurately describe the relationship between response variables and covariates.However,with the development of information technology,massive datasets are almost ubiquitous.The processing of massive data sets poses a challenge to statistical regression analysis,especially quantile regression.Usually,a single computer cannot process a complete massive dataset under single machine conditions due to limited memory.The divide and conquer algorithm is currently the main method to solve this problem and has been commonly used in massive data statistical inference.However,the non-smooth nature of quantile regression leads to the shortcomings of divide and conquer algorithm,such as large amount of calculation,when solving the problem of traditional quantile regression estimation.Therefore,the efficient algorithm of quantile regression estimation under massive data still needs further development.In this paper,we combine the divide and conquer algorithm with the smooth estimation method of quantile regression,and propose a smooth quantile regression aggregation estimation algorithm based on the estimation equation.The algorithm introduces the convolution-based kernel smoothing method of quantile regression,converts the non-smooth objective function into differentiable function,thus naturally meeting the conditions required by the aggregation estimation algorithm,and can be combined with Newton Raphson and other fast iterative algorithms to avoid annoying parameter estimation,which greatly reduces the calculation cost.Theoretical research has shown that when certain conditions are met between the number of blocks and the sample size,the aggregated estimator and the sample based estimator have the same asymptotic properties.The simulation empirical results show that the algorithm can maintain the original estimation accuracy and significantly improve the calculation speed under certain conditions.In this paper,we further extend the aggregation estimation algorithm to the nonlinear model,and propose the aggregation estimation algorithm for nonlinear smooth quantile regression under massive data by linearizing the nonlinear function.The linear approximation method makes the algorithm omit the calculation of the Hessian matrix of the nonlinear function,and further reduces the calculation cost.The simulation and empirical results demonstrate the effectiveness of the algorithm.