Abstract:Extraordinary amounts of data are being produced in many branches of science. Proven statistical methods are no longer applicable with extraordinary large datasets due to computational limitations. A critical step in big data analysis is data reduction. Existing investigations in the context of linear regression focus on subsampling-based methods. However, not only is this approach prone to sampling errors, it also leads to a covariance matrix of the estimators that is typically bounded from below by a term that is of the order of the inverse of the subdata size. We propose a novel approach, termed information-based optimal subdata selection (IBOSS). Compared to leading existing subdata methods, the IBOSS approach has the following advantages: (i) it is significantly faster; (ii) it is suitable for distributed parallel computing; (iii) the variances of the slope parameter estimators converge to 0 as the full data size increases even if the subdata size is fixed, that is, the convergence rate depends on the full data size; (iv) data analysis for IBOSS subdata is straightforward and the sampling distribution of an IBOSS estimator is easy to assess. Theoretical results and extensive simulations demonstrate that the IBOSS approach is superior to subsampling-based methods, sometimes by orders of magnitude. The advantages of the new approach are also illustrated through analysis of real data. Supplementary materials for this article are available online.

Optimal Reduced Isotonic Regression

$L_0$ Isotonic Regression With Secondary Objectives

A Genetic Algorithm for Multivariate Isotonic Regression

Best $L_p$ Isotonic Regressions, $p \in \{0, 1, \infty\}$

$L_p$ Isotonic Regression Algorithms Using an $L_0$ Approach

Nonparametric Shape-restricted Regression

Robust reduced-rank regression

An Aggressive Reduction on the Complexity of Optimization for Non-Strongly Convex Objectives

Scaled minimax optimality in high-dimensional linear regression: A non-convex algorithmic regularization approach

A joint estimation approach for monotonic regression functions in general dimensions

Low-rank Optimal Transport: Approximation, Statistics and Debiasing

An active-set based recursive approach for solving convex isotonic regression with generalized order restrictions

Some unified results on isotonic regression estimators of order restricted parameters of a general bivariate location/scale model

Improved Convergence for $\ell_\infty$ and $\ell_1$ Regression via Iteratively Reweighted Least Squares

Efficient Rank Minimization via Solving Non-convexPenalties by Iterative Shrinkage-Thresholding Algorithm

A Gradient Descent Based Algorithm for ℓ Minimization.

Information-Based Optimal Subdata Selection for Big Data Linear Regression

Optimal Subsampling for Large Sample Logistic Regression

On the Correctness of the Generalized Isotonic Recursive Partitioning Algorithm

Optimal $k$-thresholding algorithms for sparse optimization problems

From isotonic to Lipschitz regression: a new interpolative perspective on shape-restricted estimation