Optimal subsampling for large‐sample quantile regression with massive data

Li Shao,Shanshan Song,Yong Zhou

DOI: https://doi.org/10.1002/cjs.11697

2022-03-17

Canadian Journal of Statistics

Abstract:Canadian Journal of Statistics, EarlyView.

What problem does this paper attempt to address?

Optimal Subsampling for Large-Scale Quantile Regression

Mingyao Ai,Fei Wang,Jun Yu,Huiming Zhang

DOI: https://doi.org/10.1016/j.jco.2020.101512

IF: 1.333

2021-01-01

Journal of Complexity

Abstract:To deal with massive data sets, subsampling is known as an effective method which can significantly reduce computational costs in estimating model parameters. In this article, an efficient subsampling method is developed for large-scale quantile regression via Poisson sampling framework, which can solve the memory constraint problem imposed by big data. Under some mild conditions, large sample properties for the estimator involving the weak and strong consistencies, and asymptotic normality are established. Furthermore, the optimal subsampling probabilities are derived according to the A-optimality criterion. It is shown that the estimator based on the optimal subsampling asymptotically achieves a smaller variance than that by the uniform random subsampling. The proposed method is illustrated and evaluated through numerical analyses on both simulated and real data sets.
Optimal subsampling for quantile regression in big data

HaiYing Wang,Yanyuan Ma

DOI: https://doi.org/10.48550/arXiv.2001.10168

2020-01-28

Abstract:We investigate optimal subsampling for quantile regression. We derive the asymptotic distribution of a general subsampling estimator and then derive two versions of optimal subsampling probabilities. One version minimizes the trace of the asymptotic variance-covariance matrix for a linearly transformed parameter estimator and the other minimizes that of the original parameter estimator. The former does not depend on the densities of the responses given covariates and is easy to implement. Algorithms based on optimal subsampling probabilities are proposed and asymptotic distributions and asymptotic optimality of the resulting estimators are established. Furthermore, we propose an iterative subsampling procedure based on the optimal subsampling probabilities in the linearly transformed parameter estimation which has great scalability to utilize available computational resources. In addition, this procedure yields standard errors for parameter estimators without estimating the densities of the responses given the covariates. We provide numerical examples based on both simulated and real data to illustrate the proposed method.

Computation,Methodology
Optimal subsampling algorithms for composite quantile regression in massive data

Jun Jin,Shuangzhe Liu,Tiefeng Ma

DOI: https://doi.org/10.1080/02331888.2023.2239507

IF: 2.346

2023-07-26

Statistics

Abstract:Massive datasets have gained increasing prominence across various fields, but their analysis is often impeded by computational limitations. In response, Wang and Ma (Optimal subsampling for quantile regression in big data. Biometrika. 2021;108:99–112) have proposed an optimal subsampling method for quantile regression in massive datasets. Composite quantile regression, as a robust and efficient alternative to ordinary least squares regression and quantile regression in linear models, presents further complexities due to its distinct loss function. This paper extends the optimal subsampling method to accommodate composite quantile regression problems. We begin by deriving two new optimal subsampling probabilities for composite quantile regression, considering both the L- and A-optimality criteria Subsequently, we develop an adaptive two-step method based on these probabilities. The resulting estimators exhibit desirable asymptotic properties. In addition, to estimate the variance-covariance matrix without explicitly estimating the densities of the responses, we propose a combining subsamples method. Numerical studies on simulated and real data are conducted to assess and showcase the practical performance of our proposed methods.

statistics & probability
Optimal subsampling for functional composite quantile regression in massive data

Jingxiang Pan,Xiaohui Yuan,Xiaohui Yuan

DOI: https://doi.org/10.48550/arXiv.2406.19691

2024-06-28

Abstract:As computer resources become increasingly limited, traditional statistical methods face challenges in analyzing massive data, especially in functional data analysis. To address this issue, subsampling offers a viable solution by significantly reducing computational requirements. This paper introduces a subsampling technique for composite quantile regression, designed for efficient application within the functional linear model on large datasets. We establish the asymptotic distribution of the subsampling estimator and introduce an optimal subsampling method based on the functional L-optimality criterion. Results from simulation studies and the real data analysis consistently demonstrate the superiority of the L-optimality criterion-based optimal subsampling method over the uniform subsampling approach.

Methodology
Optimal subsampling for multiplicative regression with massive data

Tianzhen Wang,Haixiang Zhang

DOI: https://doi.org/10.1111/stan.12266

2022-03-21

Statistica Neerlandica

Abstract:Statistica Neerlandica, Accepted Article.

statistics & probability
Optimal subsampling algorithm for composite quantile regression with distributed data

Xiaohui Yuan,Shiting Zhou,Yue Wang

DOI: https://doi.org/10.48550/arXiv.2301.02448

2023-01-06

Abstract:For massive data stored at multiple machines, we propose a distributed subsampling procedure for the composite quantile regression. By establishing the consistency and asymptotic normality of the composite quantile regression estimator from a general subsampling algorithm, we derive the optimal subsampling probabilities and the optimal allocation sizes under the L-optimality criteria. A two-step algorithm to approximate the optimal subsampling procedure is developed. The proposed methods are illustrated through numerical experiments on simulated and real datasets.

Computation
Distributed Bootstrap Simultaneous Inference for High-Dimensional Quantile Regression

Xingcai Zhou,Zhaoyang Jing,Chao Huang

DOI: https://doi.org/10.3390/math12050735

IF: 2.4

2024-03-01

Mathematics

Abstract:Modern massive data with enormous sample size and tremendous dimensionality are usually impossible to process with a single machine. They are typically stored and processed in a distributed manner. In this paper, we propose a distributed bootstrap simultaneous inference for a high-dimensional quantile regression model using massive data. Meanwhile, a communication-efficient (CE) distributed learning algorithm is developed via the CE surrogate likelihood framework and ADMM procedure, which can handle the non-smoothness of the quantile regression loss and the Lasso penalty. We theoretically prove the convergence of the algorithm and establish a lower bound on the number of communication rounds ιmin that warrant statistical accuracy and efficiency. The distributed bootstrap validity and efficiency are corroborated by an extensive simulation study.

mathematics
Optimal subsampling for functional quantile regression

Qian Yan,Hanyu Li,Chengmei Niu

DOI: https://doi.org/10.48550/arXiv.2205.02718

2022-05-05

Abstract:Subsampling is an efficient method to deal with massive data. In this paper, we investigate the optimal subsampling for linear quantile regression when the covariates are functions. The asymptotic distribution of the subsampling estimator is first derived. Then, we obtain the optimal subsampling probabilities based on the A-optimality criterion. Furthermore, the modified subsampling probabilities without estimating the densities of the response variables given the covariates are also proposed, which are easier to implement in practise. Numerical experiments on synthetic and real data show that the proposed methods always outperform the one with uniform sampling and can approximate the results based on full data well with less computational efforts.

Numerical Analysis,Statistics Theory
Optimal subsampling for composite quantile regression in big data

Xiaohui Yuan,Yong Li,Xiaogang Dong,Tianqing Liu

DOI: https://doi.org/10.1007/s00362-022-01292-1

2022-02-08

Statistical Papers

Abstract:The composite quantile regression (CQR) is an efficient and robust alternative to the least squares for estimating regression coefficients in a linear model. We investigate optimal subsampling for CQR with massive datasets. By establishing the consistency and asymptotic normality of the CQR estimator from a general subsampling algorithm, we derive the optimal subsampling probabilities under the L- and A-optimality criteria. The L-optimality criterion minimizes the trace of the asymptotic variance–covariance matrix of the estimator for a linearly transformed regression parameters and the A-optimality criterion minimizes that of the estimator for regression parameters. The L-optimal subsampling probabilities is easy to implement as they do not depend on the densities of the responses given covariates. Based on the L-optimal subsampling probabilities, we propose algorithms for computing the resulting estimators and their asymptotic distributions and asymptotic optimality are established. To obtain standard errors for CQR estimators without estimating the densities of the responses given the covariates, we propose an iterative subsampling procedure based on the L-optimal subsampling probabilities. The proposed methods are illustrated through numerical experiments on simulated and real datasets.

statistics & probability
Optimal Subsampling Approaches for Large Sample Linear Regression

Rong Zhu,Ping Ma,Michael W. Mahoney,Bin Yu

DOI: https://doi.org/10.48550/arXiv.1509.05111

2015-11-23

Abstract:A significant hurdle for analyzing large sample data is the lack of effective statistical computing and inference methods. An emerging powerful approach for analyzing large sample data is subsampling, by which one takes a random subsample from the original full sample and uses it as a surrogate for subsequent computation and estimation. In this paper, we study subsampling methods under two scenarios: approximating the full sample ordinary least-square (OLS) estimator and estimating the coefficients in linear regression. We present two algorithms, weighted estimation algorithm and unweighted estimation algorithm, and analyze asymptotic behaviors of their resulting subsample estimators under general conditions. For the weighted estimation algorithm, we propose a criterion for selecting the optimal sampling probability by making use of the asymptotic results. On the basis of the criterion, we provide two novel subsampling methods, the optimal subsampling and the predictor- length subsampling methods. The predictor-length subsampling method is based on the L2 norm of predictors rather than leverage scores. Its computational cost is scalable. For unweighted estimation algorithm, we show that its resulting subsample estimator is not consistent to the full sample OLS estimator. However, it has better performance than the weighted estimation algorithm for estimating the coefficients. Simulation studies and a real data example are used to demonstrate the effectiveness of our proposed subsampling methods.

Methodology,Machine Learning
Optimal Subsampling for Large Sample Logistic Regression

HaiYing Wang,Rong Zhu,Ping Ma

DOI: https://doi.org/10.1080/01621459.2017.1292914

2018-03-08

Abstract:For massive data, the family of subsampling algorithms is popular to downsize the data volume and reduce computational burden. Existing studies focus on approximating the ordinary least squares estimate in linear regression, where statistical leverage scores are often used to define subsampling probabilities. In this paper, we propose fast subsampling algorithms to efficiently approximate the maximum likelihood estimate in logistic regression. We first establish consistency and asymptotic normality of the estimator from a general subsampling algorithm, and then derive optimal subsampling probabilities that minimize the asymptotic mean squared error of the resultant estimator. An alternative minimization criterion is also proposed to further reduce the computational cost. The optimal subsampling probabilities depend on the full data estimate, so we develop a two-step algorithm to approximate the optimal subsampling procedure. This algorithm is computationally efficient and has a significant reduction in computing time compared to the full data approach. Consistency and asymptotic normality of the estimator from a two-step algorithm are also established. Synthetic and real data sets are used to evaluate the practical performance of the proposed method.

Computation,Methodology,Machine Learning
Optimal subsampling algorithm for the marginal model with large longitudinal data

Haohui Han,Liya Fu

DOI: https://doi.org/10.48550/arXiv.2311.08812

2023-11-15

Abstract:Big data is ubiquitous in practices, and it has also led to heavy computation burden. To reduce the calculation cost and ensure the effectiveness of parameter estimators, an optimal subset sampling method is proposed to estimate the parameters in marginal models with massive longitudinal data. The optimal subsampling probabilities are derived, and the corresponding asymptotic properties are established to ensure the consistency and asymptotic normality of the estimator. Extensive simulation studies are carried out to evaluate the performance of the proposed method for continuous, binary and count data and with four different working correlation matrices. A depression data is used to illustrate the proposed method.

Methodology
Distributed quantile regression for massive heterogeneous data

Aijun Hu,Yuling Jiao,Yanyan Liu,Yueyong Shi,Yuanshan Wu

DOI: https://doi.org/10.1016/j.neucom.2021.03.041

IF: 6

2021-01-01

Neurocomputing

Abstract:Massive data sets pose great challenges to data analysis because of their heterogeneous data structure and limited computer memory. Jordan et al. (2019, Journal of American Statistical Association) has proposed a communication-efficient surrogate likelihood (CSL) method to solve distributed learning problems. However, their method cannot be directly applied to quantile regression because the loss function in quantile regression does not meet the smoothness requirement in CSL method. In this paper, we extend CSL method so that it is applicable to quantile regression problems. The key idea is to construct a surrogate loss function which relates to the local data only through subgradients of the loss function. The alternating direction method of multipliers (ADMM) algorithm is used to address computational issues caused by the non-smooth loss function. Our theoretical analysis establishes the consistency and asymptotic normality for the proposed method. Simulation studies and applications to real data show that our method works well. (c) 2021 Elsevier B.V. All rights reserved.
Online Quantile Regression

Yinan Shen,Dong Xia,Wen-Xin Zhou

2024-02-19

Abstract:This paper addresses the challenge of integrating sequentially arriving data within the quantile regression framework, where the number of features is allowed to grow with the number of observations, the horizon is unknown, and memory is limited. We employ stochastic sub-gradient descent to minimize the empirical check loss and study its statistical properties and regret performance. In our analysis, we unveil the delicate interplay between updating iterates based on individual observations versus batches of observations, revealing distinct regularity properties in each scenario. Our method ensures long-term optimal estimation irrespective of the chosen update strategy. Importantly, our contributions go beyond prior works by achieving exponential-type concentration inequalities and attaining optimal regret and error rates that exhibit only \textsf{ short-term} sensitivity to initial errors. A key insight from our study is the delicate statistical analyses and the revelation that appropriate stepsize schemes significantly mitigate the impact of initial errors on subsequent errors and regrets. This underscores the robustness of stochastic sub-gradient descent in handling initial uncertainties, emphasizing its efficacy in scenarios where the sequential arrival of data introduces uncertainties regarding both the horizon and the total number of observations. Additionally, when the initial error rate is well-controlled, there is a trade-off between short-term error rate and long-term optimality. Due to the lack of delicate statistical analysis for squared loss, we also briefly discuss its properties and proper schemes. Extensive simulations support our theoretical findings.

Statistics Theory,Information Theory,Methodology
Optimal Subsampling Algorithms for Big Data Regressions

Mingyao Ai,Jun Yu,Huiming Zhang,HaiYing Wang

DOI: https://doi.org/10.5705/ss.202018.0439

2019-06-26

Abstract:To fast approximate maximum likelihood estimators with massive data, this paper studies the Optimal Subsampling Method under the A-optimality Criterion (OSMAC) for generalized linear models. The consistency and asymptotic normality of the estimator from a general subsampling algorithm are established, and optimal subsampling probabilities under the A- and L-optimality criteria are derived. Furthermore, using Frobenius norm matrix concentration inequalities, finite sample properties of the subsample estimator based on optimal subsampling probabilities are also derived. Since the optimal subsampling probabilities depend on the full data estimate, an adaptive two-step algorithm is developed. Asymptotic normality and optimality of the estimator from this adaptive algorithm are established. The proposed methods are illustrated and evaluated through numerical experiments on simulated and real datasets.

Methodology,Statistics Theory,Computation
Optimal Distributed Subsampling for Maximum Quasi-Likelihood Estimators with Massive Data

Jun Yu,HaiYing Wang,Mingyao Ai,Huiming Zhang

DOI: https://doi.org/10.48550/arXiv.2005.10435

2021-07-05

Abstract:Nonuniform subsampling methods are effective to reduce computational burden and maintain estimation efficiency for massive data. Existing methods mostly focus on subsampling with replacement due to its high computational efficiency. If the data volume is so large that nonuniform subsampling probabilities cannot be calculated all at once, then subsampling with replacement is infeasible to implement. This paper solves this problem using Poisson subsampling. We first derive optimal Poisson subsampling probabilities in the context of quasi-likelihood estimation under the A- and L-optimality criteria. For a practically implementable algorithm with approximated optimal subsampling probabilities, we establish the consistency and asymptotic normality of the resultant estimators. To deal with the situation that the full data are stored in different blocks or at multiple locations, we develop a distributed subsampling framework, in which statistics are computed simultaneously on smaller partitions of the full data. Asymptotic properties of the resultant aggregated estimator are investigated. We illustrate and evaluate the proposed strategies through numerical experiments on simulated and real data sets.

Methodology,Distributed, Parallel, and Cluster Computing,Computation,Machine Learning
Optimal Subsampling Bootstrap for Massive Data

Yingying Ma,Chenlei Leng,Hansheng Wang

DOI: https://doi.org/10.1080/07350015.2023.2166514

2024-01-01

Abstract:The bootstrap is a widely used procedure for statistical inference because of its simplicity and attractive statistical properties. However, the vanilla version of bootstrap is no longer feasible computationally for many modern massive datasets due to the need to repeatedly resample the entire data. Therefore, several improvements to the bootstrap method have been made in recent years, which assess the quality of estimators by subsampling the full dataset before resampling the subsamples. Naturally, the performance of these modern subsampling methods is influenced by tuning parameters such as the size of subsamples, the number of subsamples, and the number of resamples per subsample. In this article, we develop a novel hyperparameter selection methodology for selecting these tuning parameters. Formulated as an optimization problem to find the optimal value of some measure of accuracy of an estimator subject to computational cost, our framework provides closed-form solutions for the optimal hyperparameter values for subsampled bootstrap, subsampled double bootstrap and bag of little bootstraps, at no or little extra time cost. Using the mean square errors as a proxy of the accuracy measure, we apply our methodology to study, compare and improve the performance of these modern versions of bootstrap developed for massive data through numerical study. The results are promising.
Estimation and testing of expectile regression with efficient subsampling for massive data

Baolin Chen,Shanshan Song,Yong Zhou

DOI: https://doi.org/10.1007/s00362-024-01571-z

2024-09-30

Statistical Papers

Abstract:Subsampling strategy plays a crucial role in statistical inference for massive data owing to its computing and storage superiority. The parameter estimation and hypothesis testing of expectile regression for massive data is of concern. This paper offers an alternative to the traditional asymmetric least square (ALS) estimator via smooth approximation of loss function. Then, an efficient subsampling algorithm based on Newton's iteration is proposed. We prove consistency and asymptotic normality and provide the optimal subsampling probability and the proper order of smoothing parameter. We also apply the subsampling strategy for hypothesis testing, where the proposed test statistics have bigger power, compared with the test statistic based on the simple random subsampling. Simulation and two real data examples demonstrate the effectiveness of the proposed subsampling estimation and testing methods.

statistics & probability
Sample size determination for multidimensional parameters and the A-optimal subsampling in a big data linear regression model

Sheng Zhang,Fei Tan,Hanxiang Peng

DOI: https://doi.org/10.1080/00949655.2024.2434669

IF: 1.225

2024-12-03

Journal of Statistical Computation and Simulation

Abstract:To efficiently approximate the least squares estimator (LSE) in a Big Data linear regression model using a subsampling approach, optimal sampling distributions were derived by minimizing the trace norm of the covariance matrix of a smooth function of the subsampling LSE. An algorithm was developed that significantly reduces the computation time for the subsampling LSE compared to the full-sample LSE. Additionally, the subsampling LSE was shown to be asymptotically normal almost surely for an arbitrary sampling distribution under suitable conditions. Motivated by the need for subsampling in Big Data analysis and data splitting in machine learning, we investigated sample size determination (SSD) for multidimensional parameters and derived analytical formulas for calculating sample sizes. Through extensive simulations and real-world data applications, we assessed the numerical properties of both the subsampling approach and SSD methodology. Our findings revealed that the A-optimal subsampling method significantly outperformed uniform and leverage-score subsampling techniques. Furthermore, the algorithm considerably reduced the computational time required for implementing the full sample LSE. Additionally, the SSD provided a theoretical basis for selecting sample sizes.

statistics & probability,computer science, interdisciplinary applications
Optimal subsampling for modal regression in massive data

Yue Chao,Lei Huang,Xuejun Ma,Jiajun Sun

DOI: https://doi.org/10.1007/s00184-023-00916-2

IF: 0.96

2023-06-29

Metrika

Abstract:Many modern statistical analysis research efforts are focused on solving the limited computational resources problem that arises when dealing with large datasets. One popular and effective method to address this challenge is to obtain informative subdata from the full dataset based on optimal subsampling probabilities. In this article, we present an optimal subsampling approach for big data modal regression from the perspective of minimizing asymptotic mean squared error. The estimation procedure is achieved by running a two-step algorithm based on the modal expectation-maximization algorithm when the bandwidth for the modal regression is not related to the subsample size. Under certain regularity conditions, we investigate the consistency and asymptotic normality of the subsample-based estimator given the full data. Furthermore, an optimal bandwidth selection approach within this framework is also investigated. Simulation studies demonstrate that our proposed subsampling method performs well in the context of big data modal regression. Empirical evaluation is also conducted using real data.

statistics & probability

Optimal subsampling for large‐sample quantile regression with massive data

Optimal Subsampling for Large-Scale Quantile Regression

Optimal subsampling for quantile regression in big data

Optimal subsampling algorithms for composite quantile regression in massive data

Optimal subsampling for functional composite quantile regression in massive data

Optimal subsampling for multiplicative regression with massive data

Optimal subsampling algorithm for composite quantile regression with distributed data

Distributed Bootstrap Simultaneous Inference for High-Dimensional Quantile Regression

Optimal subsampling for functional quantile regression

Optimal subsampling for composite quantile regression in big data

Optimal Subsampling Approaches for Large Sample Linear Regression

Optimal Subsampling for Large Sample Logistic Regression

Optimal subsampling algorithm for the marginal model with large longitudinal data

Distributed quantile regression for massive heterogeneous data

Online Quantile Regression

Optimal Subsampling Algorithms for Big Data Regressions

Optimal Distributed Subsampling for Maximum Quasi-Likelihood Estimators with Massive Data

Optimal Subsampling Bootstrap for Massive Data

Estimation and testing of expectile regression with efficient subsampling for massive data

Sample size determination for multidimensional parameters and the A-optimal subsampling in a big data linear regression model

Optimal subsampling for modal regression in massive data