Optimal subsampling for multiplicative regression with massive data

Tianzhen Wang,Haixiang Zhang

DOI: https://doi.org/10.1111/stan.12266

2022-03-21

Statistica Neerlandica

Abstract:Statistica Neerlandica, Accepted Article.

statistics & probability

What problem does this paper attempt to address?

Optimal subsampling for large‐sample quantile regression with massive data

Li Shao,Shanshan Song,Yong Zhou

DOI: https://doi.org/10.1002/cjs.11697

2022-03-17

Canadian Journal of Statistics

Abstract:Canadian Journal of Statistics, EarlyView.
Optimal Subsampling for Large Sample Logistic Regression

HaiYing Wang,Rong Zhu,Ping Ma

DOI: https://doi.org/10.1080/01621459.2017.1292914

2018-03-08

Abstract:For massive data, the family of subsampling algorithms is popular to downsize the data volume and reduce computational burden. Existing studies focus on approximating the ordinary least squares estimate in linear regression, where statistical leverage scores are often used to define subsampling probabilities. In this paper, we propose fast subsampling algorithms to efficiently approximate the maximum likelihood estimate in logistic regression. We first establish consistency and asymptotic normality of the estimator from a general subsampling algorithm, and then derive optimal subsampling probabilities that minimize the asymptotic mean squared error of the resultant estimator. An alternative minimization criterion is also proposed to further reduce the computational cost. The optimal subsampling probabilities depend on the full data estimate, so we develop a two-step algorithm to approximate the optimal subsampling procedure. This algorithm is computationally efficient and has a significant reduction in computing time compared to the full data approach. Consistency and asymptotic normality of the estimator from a two-step algorithm are also established. Synthetic and real data sets are used to evaluate the practical performance of the proposed method.

Computation,Methodology,Machine Learning
Optimal Subsampling for Large-Scale Quantile Regression

Mingyao Ai,Fei Wang,Jun Yu,Huiming Zhang

DOI: https://doi.org/10.1016/j.jco.2020.101512

IF: 1.333

2021-01-01

Journal of Complexity

Abstract:To deal with massive data sets, subsampling is known as an effective method which can significantly reduce computational costs in estimating model parameters. In this article, an efficient subsampling method is developed for large-scale quantile regression via Poisson sampling framework, which can solve the memory constraint problem imposed by big data. Under some mild conditions, large sample properties for the estimator involving the weak and strong consistencies, and asymptotic normality are established. Furthermore, the optimal subsampling probabilities are derived according to the A-optimality criterion. It is shown that the estimator based on the optimal subsampling asymptotically achieves a smaller variance than that by the uniform random subsampling. The proposed method is illustrated and evaluated through numerical analyses on both simulated and real data sets.
Optimal Subsampling Algorithms for Big Data Regressions

Mingyao Ai,Jun Yu,Huiming Zhang,HaiYing Wang

DOI: https://doi.org/10.5705/ss.202018.0439

2019-06-26

Abstract:To fast approximate maximum likelihood estimators with massive data, this paper studies the Optimal Subsampling Method under the A-optimality Criterion (OSMAC) for generalized linear models. The consistency and asymptotic normality of the estimator from a general subsampling algorithm are established, and optimal subsampling probabilities under the A- and L-optimality criteria are derived. Furthermore, using Frobenius norm matrix concentration inequalities, finite sample properties of the subsample estimator based on optimal subsampling probabilities are also derived. Since the optimal subsampling probabilities depend on the full data estimate, an adaptive two-step algorithm is developed. Asymptotic normality and optimality of the estimator from this adaptive algorithm are established. The proposed methods are illustrated and evaluated through numerical experiments on simulated and real datasets.

Methodology,Statistics Theory,Computation
Optimal subsampling algorithm for the marginal model with large longitudinal data

Haohui Han,Liya Fu

DOI: https://doi.org/10.48550/arXiv.2311.08812

2023-11-15

Abstract:Big data is ubiquitous in practices, and it has also led to heavy computation burden. To reduce the calculation cost and ensure the effectiveness of parameter estimators, an optimal subset sampling method is proposed to estimate the parameters in marginal models with massive longitudinal data. The optimal subsampling probabilities are derived, and the corresponding asymptotic properties are established to ensure the consistency and asymptotic normality of the estimator. Extensive simulation studies are carried out to evaluate the performance of the proposed method for continuous, binary and count data and with four different working correlation matrices. A depression data is used to illustrate the proposed method.

Methodology
Optimal subsampling algorithms for composite quantile regression in massive data

Jun Jin,Shuangzhe Liu,Tiefeng Ma

DOI: https://doi.org/10.1080/02331888.2023.2239507

IF: 2.346

2023-07-26

Statistics

Abstract:Massive datasets have gained increasing prominence across various fields, but their analysis is often impeded by computational limitations. In response, Wang and Ma (Optimal subsampling for quantile regression in big data. Biometrika. 2021;108:99–112) have proposed an optimal subsampling method for quantile regression in massive datasets. Composite quantile regression, as a robust and efficient alternative to ordinary least squares regression and quantile regression in linear models, presents further complexities due to its distinct loss function. This paper extends the optimal subsampling method to accommodate composite quantile regression problems. We begin by deriving two new optimal subsampling probabilities for composite quantile regression, considering both the L- and A-optimality criteria Subsequently, we develop an adaptive two-step method based on these probabilities. The resulting estimators exhibit desirable asymptotic properties. In addition, to estimate the variance-covariance matrix without explicitly estimating the densities of the responses, we propose a combining subsamples method. Numerical studies on simulated and real data are conducted to assess and showcase the practical performance of our proposed methods.

statistics & probability
Optimal subsampling for quantile regression in big data

HaiYing Wang,Yanyuan Ma

DOI: https://doi.org/10.48550/arXiv.2001.10168

2020-01-28

Abstract:We investigate optimal subsampling for quantile regression. We derive the asymptotic distribution of a general subsampling estimator and then derive two versions of optimal subsampling probabilities. One version minimizes the trace of the asymptotic variance-covariance matrix for a linearly transformed parameter estimator and the other minimizes that of the original parameter estimator. The former does not depend on the densities of the responses given covariates and is easy to implement. Algorithms based on optimal subsampling probabilities are proposed and asymptotic distributions and asymptotic optimality of the resulting estimators are established. Furthermore, we propose an iterative subsampling procedure based on the optimal subsampling probabilities in the linearly transformed parameter estimation which has great scalability to utilize available computational resources. In addition, this procedure yields standard errors for parameter estimators without estimating the densities of the responses given the covariates. We provide numerical examples based on both simulated and real data to illustrate the proposed method.

Computation,Methodology
Optimal subsampling for modal regression in massive data

Yue Chao,Lei Huang,Xuejun Ma,Jiajun Sun

DOI: https://doi.org/10.1007/s00184-023-00916-2

IF: 0.96

2023-06-29

Metrika

Abstract:Many modern statistical analysis research efforts are focused on solving the limited computational resources problem that arises when dealing with large datasets. One popular and effective method to address this challenge is to obtain informative subdata from the full dataset based on optimal subsampling probabilities. In this article, we present an optimal subsampling approach for big data modal regression from the perspective of minimizing asymptotic mean squared error. The estimation procedure is achieved by running a two-step algorithm based on the modal expectation-maximization algorithm when the bandwidth for the modal regression is not related to the subsample size. Under certain regularity conditions, we investigate the consistency and asymptotic normality of the subsample-based estimator given the full data. Furthermore, an optimal bandwidth selection approach within this framework is also investigated. Simulation studies demonstrate that our proposed subsampling method performs well in the context of big data modal regression. Empirical evaluation is also conducted using real data.

statistics & probability
Sample size determination for multidimensional parameters and the A-optimal subsampling in a big data linear regression model

Sheng Zhang,Fei Tan,Hanxiang Peng

DOI: https://doi.org/10.1080/00949655.2024.2434669

IF: 1.225

2024-12-03

Journal of Statistical Computation and Simulation

Abstract:To efficiently approximate the least squares estimator (LSE) in a Big Data linear regression model using a subsampling approach, optimal sampling distributions were derived by minimizing the trace norm of the covariance matrix of a smooth function of the subsampling LSE. An algorithm was developed that significantly reduces the computation time for the subsampling LSE compared to the full-sample LSE. Additionally, the subsampling LSE was shown to be asymptotically normal almost surely for an arbitrary sampling distribution under suitable conditions. Motivated by the need for subsampling in Big Data analysis and data splitting in machine learning, we investigated sample size determination (SSD) for multidimensional parameters and derived analytical formulas for calculating sample sizes. Through extensive simulations and real-world data applications, we assessed the numerical properties of both the subsampling approach and SSD methodology. Our findings revealed that the A-optimal subsampling method significantly outperformed uniform and leverage-score subsampling techniques. Furthermore, the algorithm considerably reduced the computational time required for implementing the full sample LSE. Additionally, the SSD provided a theoretical basis for selecting sample sizes.

statistics & probability,computer science, interdisciplinary applications
Robust and efficient subsampling algorithms for massive data logistic regression

Jun Jin,Shuangzhe Liu,Tiefeng Ma

DOI: https://doi.org/10.1080/02664763.2023.2205611

2023-04-26

Abstract:Datasets that are big with regard to their volume, variety and velocity are becoming increasingly common. However, limitations in computer processing can often restrict analysis performed on them. Nonuniform subsampling methods are effective in reducing computational loads for massive data. However, the variance of the estimator of nonuniform subsampling methods becomes large when the subsampling probabilities are highly heterogenous. To this end, we develop two new algorithms to improve the estimation method for massive data logistic regression based on a chosen hard threshold value and combining subsamples, respectively. The basic idea of the hard threshold method is to carefully select a threshold value and then replace subsampling probabilities lower than the threshold value with the chosen value itself. The main idea behind the combining subsamples method is to better exploit information in the data without hitting the computation bottleneck by generating many subsamples and then combining estimates constructed from the subsamples. The combining subsamples method obtains the standard error of the parameter estimator without estimating the sandwich matrix, which provides convenience for statistical inference in massive data, and can significantly improve the estimation efficiency. Asymptotic properties of the resultant estimators are established. Simulations and analysis of real data are conducted to assess and showcase the practical performance of the proposed methods.
Optimal subsampling for functional composite quantile regression in massive data

Jingxiang Pan,Xiaohui Yuan,Xiaohui Yuan

DOI: https://doi.org/10.48550/arXiv.2406.19691

2024-06-28

Abstract:As computer resources become increasingly limited, traditional statistical methods face challenges in analyzing massive data, especially in functional data analysis. To address this issue, subsampling offers a viable solution by significantly reducing computational requirements. This paper introduces a subsampling technique for composite quantile regression, designed for efficient application within the functional linear model on large datasets. We establish the asymptotic distribution of the subsampling estimator and introduce an optimal subsampling method based on the functional L-optimality criterion. Results from simulation studies and the real data analysis consistently demonstrate the superiority of the L-optimality criterion-based optimal subsampling method over the uniform subsampling approach.

Methodology
Optimal Subsampling Approaches for Large Sample Linear Regression

Rong Zhu,Ping Ma,Michael W. Mahoney,Bin Yu

DOI: https://doi.org/10.48550/arXiv.1509.05111

2015-11-23

Abstract:A significant hurdle for analyzing large sample data is the lack of effective statistical computing and inference methods. An emerging powerful approach for analyzing large sample data is subsampling, by which one takes a random subsample from the original full sample and uses it as a surrogate for subsequent computation and estimation. In this paper, we study subsampling methods under two scenarios: approximating the full sample ordinary least-square (OLS) estimator and estimating the coefficients in linear regression. We present two algorithms, weighted estimation algorithm and unweighted estimation algorithm, and analyze asymptotic behaviors of their resulting subsample estimators under general conditions. For the weighted estimation algorithm, we propose a criterion for selecting the optimal sampling probability by making use of the asymptotic results. On the basis of the criterion, we provide two novel subsampling methods, the optimal subsampling and the predictor- length subsampling methods. The predictor-length subsampling method is based on the L2 norm of predictors rather than leverage scores. Its computational cost is scalable. For unweighted estimation algorithm, we show that its resulting subsample estimator is not consistent to the full sample OLS estimator. However, it has better performance than the weighted estimation algorithm for estimating the coefficients. Simulation studies and a real data example are used to demonstrate the effectiveness of our proposed subsampling methods.

Methodology,Machine Learning
Subsampling for Big Data Linear Models with Measurement Errors

Jiangshan Ju,Mingqiu Wang,Shengli Zhao

2024-09-19

Abstract:Subsampling algorithms for various parametric regression models with massive data have been extensively investigated in recent years. However, all existing studies on subsampling heavily rely on clean massive data. In practical applications, the observed covariates may suffer from inaccuracies due to measurement errors. To address the challenge of large datasets with measurement errors, this study explores two subsampling algorithms based on the corrected likelihood approach: the optimal subsampling algorithm utilizing inverse probability weighting and the perturbation subsampling algorithm employing random weighting assuming a perfectly known distribution. Theoretical properties for both algorithms are provided. Numerical simulations and two real-world examples demonstrate the effectiveness of these proposed methods compared to other uncorrected algorithms.

Statistics Theory
Estimation and testing of expectile regression with efficient subsampling for massive data

Baolin Chen,Shanshan Song,Yong Zhou

DOI: https://doi.org/10.1007/s00362-024-01571-z

2024-09-30

Statistical Papers

Abstract:Subsampling strategy plays a crucial role in statistical inference for massive data owing to its computing and storage superiority. The parameter estimation and hypothesis testing of expectile regression for massive data is of concern. This paper offers an alternative to the traditional asymmetric least square (ALS) estimator via smooth approximation of loss function. Then, an efficient subsampling algorithm based on Newton's iteration is proposed. We prove consistency and asymptotic normality and provide the optimal subsampling probability and the proper order of smoothing parameter. We also apply the subsampling strategy for hypothesis testing, where the proposed test statistics have bigger power, compared with the test statistic based on the simple random subsampling. Simulation and two real data examples demonstrate the effectiveness of the proposed subsampling estimation and testing methods.

statistics & probability
Optimal subsampling for high-dimensional ridge regression

Hanyu Li,Chengmei Niu

DOI: https://doi.org/10.1016/j.knosys.2024.111426

IF: 8.139

2024-02-01

Knowledge-Based Systems

Abstract:We investigate the feature compression of high-dimensional ridge regression using the optimal subsampling technique. Specifically, based on the basic framework of random sampling algorithm on feature for ridge regression and the A-optimal design criterion, we first obtain a set of optimal subsampling probabilities. Considering that the obtained probabilities are uneconomical, we then propose the nearly optimal ones. With these probabilities, a two step iterative algorithm is established which has lower computational cost and higher accuracy. We provide theoretical analysis and numerical experiments to support the proposed methods. Numerical results demonstrate the decent performance of our methods.

computer science, artificial intelligence
Optimal Subsampling for Large Sample Ridge Regression

CHEN Yunlu,ZHANG Nan

DOI: https://doi.org/10.48550/arxiv.2204.04776

2022-01-01

Abstract: Subsampling is a popular approach to alleviating the computational burden for analyzing massive datasets. Recent efforts have been devoted to various statistical models without explicit regularization. In this paper, we develop an efficient subsampling procedure for the large sample linear ridge regression. In contrast to the ordinary least square estimator, the introduction of the ridge penalty leads to a subtle trade-off between bias and variance. We first investigate the asymptotic properties of the subsampling estimator and then propose to minimize the asymptotic-mean-squared-error criterion for optimality. The resulting subsampling probability involves both ridge leverage score and L2 norm of the predictor. To further reduce the computational cost for calculating the ridge leverage scores, we propose the algorithm with efficient approximation. We show by synthetic and real datasets that the algorithm is both statistically accurate and computationally efficient compared with existing subsampling based methods.
Poisson Regression in one Covariate on Massive Data

Torsten Reuter,Rainer Schwabe

2024-03-27

Abstract:The goal of subsampling is to select an informative subset of all observations, when using the full data for statistical analysis is not viable. We construct locally $ D $-optimal subsampling designs under a Poisson regression model with a log link in one covariate. A Representation of the support of locally $ D $-optimal subsampling designs is established. We make statements on scale-location transformations of the covariate that require a simultaneous transformation of the regression parameter. The performance of the methods is demonstrated by illustrating examples. To show the advantage of the optimal subsampling designs, we examine the efficiency of uniform random subsampling as well as of two heuristic designs. Further, the efficiency of locally $ D $-optimal subsampling designs is studied when the parameter is misspecified.

Statistics Theory
Poisson subsampling-based estimation for growing-dimensional expectile regression in massive data

Xiaoyan Li,Xiaochao Xia,Zhimin Zhang

DOI: https://doi.org/10.1007/s11222-024-10449-x

IF: 2.3241

2024-06-16

Statistics and Computing

Abstract:As an effective tool for data analysis, expectile regression is widely used in the fields of statistics, econometrics and finance. However, most studies focus on the case where the sample size is not massive and the dimension is low or fixed. This paper studies the parameter estimation and inference for large-scale expectile regression when the number of parameters grows to infinity. Specifically, an inverse probability weighted asymmetric least squares estimator based on Poisson subsampling (ALS-P) is proposed. Theoretically, the convergence rate and asymptotic normality for ALS-P are established. Furthermore, the optimal subsampling probabilities based on the L-optimality criterion are derived. Finally, extensive simulations and two real-world datasets are conducted to illustrate the effectiveness of the proposed methods.

statistics & probability,computer science, theory & methods
Optimal subsampling algorithm for composite quantile regression with distributed data

Xiaohui Yuan,Shiting Zhou,Yue Wang

DOI: https://doi.org/10.48550/arXiv.2301.02448

2023-01-06

Abstract:For massive data stored at multiple machines, we propose a distributed subsampling procedure for the composite quantile regression. By establishing the consistency and asymptotic normality of the composite quantile regression estimator from a general subsampling algorithm, we derive the optimal subsampling probabilities and the optimal allocation sizes under the L-optimality criteria. A two-step algorithm to approximate the optimal subsampling procedure is developed. The proposed methods are illustrated through numerical experiments on simulated and real datasets.

Computation
Optimal Subsampling Bootstrap for Massive Data

Yingying Ma,Chenlei Leng,Hansheng Wang

DOI: https://doi.org/10.1080/07350015.2023.2166514

2024-01-01

Abstract:The bootstrap is a widely used procedure for statistical inference because of its simplicity and attractive statistical properties. However, the vanilla version of bootstrap is no longer feasible computationally for many modern massive datasets due to the need to repeatedly resample the entire data. Therefore, several improvements to the bootstrap method have been made in recent years, which assess the quality of estimators by subsampling the full dataset before resampling the subsamples. Naturally, the performance of these modern subsampling methods is influenced by tuning parameters such as the size of subsamples, the number of subsamples, and the number of resamples per subsample. In this article, we develop a novel hyperparameter selection methodology for selecting these tuning parameters. Formulated as an optimization problem to find the optimal value of some measure of accuracy of an estimator subject to computational cost, our framework provides closed-form solutions for the optimal hyperparameter values for subsampled bootstrap, subsampled double bootstrap and bag of little bootstraps, at no or little extra time cost. Using the mean square errors as a proxy of the accuracy measure, we apply our methodology to study, compare and improve the performance of these modern versions of bootstrap developed for massive data through numerical study. The results are promising.

Optimal subsampling for multiplicative regression with massive data

Optimal subsampling for large‐sample quantile regression with massive data

Optimal Subsampling for Large Sample Logistic Regression

Optimal Subsampling for Large-Scale Quantile Regression

Optimal Subsampling Algorithms for Big Data Regressions

Optimal subsampling algorithm for the marginal model with large longitudinal data

Optimal subsampling algorithms for composite quantile regression in massive data

Optimal subsampling for quantile regression in big data

Optimal subsampling for modal regression in massive data

Sample size determination for multidimensional parameters and the A-optimal subsampling in a big data linear regression model

Robust and efficient subsampling algorithms for massive data logistic regression

Optimal subsampling for functional composite quantile regression in massive data

Optimal Subsampling Approaches for Large Sample Linear Regression

Subsampling for Big Data Linear Models with Measurement Errors

Estimation and testing of expectile regression with efficient subsampling for massive data

Optimal subsampling for high-dimensional ridge regression

Optimal Subsampling for Large Sample Ridge Regression

Poisson Regression in one Covariate on Massive Data

Poisson subsampling-based estimation for growing-dimensional expectile regression in massive data

Optimal subsampling algorithm for composite quantile regression with distributed data

Optimal Subsampling Bootstrap for Massive Data