Abstract:We characterize the squared prediction risk of ensemble estimators obtained through subagging (subsample bootstrap aggregating) regularized M-estimators and construct a consistent estimator for the risk. Specifically, we consider a heterogeneous collection of $M \ge 1$ regularized M-estimators, each trained with (possibly different) subsample sizes, convex differentiable losses, and convex regularizers. We operate under the proportional asymptotics regime, where the sample size $n$, feature size $p$, and subsample sizes $k_m$ for $m \in [M]$ all diverge with fixed limiting ratios $n/p$ and $k_m/n$. Key to our analysis is a new result on the joint asymptotic behavior of correlations between the estimator and residual errors on overlapping subsamples, governed through a (provably) contractible nonlinear system of equations. Of independent interest, we also establish convergence of trace functionals related to degrees of freedom in the non-ensemble setting (with $M = 1$) along the way, extending previously known cases for square loss and ridge, lasso regularizers. When specialized to homogeneous ensembles trained with a common loss, regularizer, and subsample size, the risk characterization sheds some light on the implicit regularization effect due to the ensemble and subsample sizes $(M,k)$. For any ensemble size $M$, optimally tuning subsample size yields sample-wise monotonic risk. For the full-ensemble estimator (when $M \to \infty$), the optimal subsample size $k^\star$ tends to be in the overparameterized regime $(k^\star \le \min\{n,p\})$, when explicit regularization is vanishing. Finally, joint optimization of subsample size, ensemble size, and regularization can significantly outperform regularizer optimization alone on the full data (without any subagging).

Asymptotically efficient estimators for tail probabilities of extremals of $β$-Jacobi ensembles

Importance Sampling for the Extremal Eigenvalue of $β$-Jacobi ensemble

Approximation of the Tail Probability of Randomly Weighted Sums and Applications

Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity

Conditional Tail-Related Risk Estimation Using Composite Asymmetric Least Squares and Empirical Likelihood

Precise large deviations for randomly weighted sums of negatively dependent random variables with consistently varying tails

Optimal tail estimates in $β$-ensembles and applications to last passage percolation

On Beta-Product Convolutions

Prediction of Extremal Expectile Based on Regression Models With Heteroscedastic Extremes

Beta Jacobi ensembles and associated Jacobi polynomials

Fast sampling from $β$-ensembles

Large Deviations for Weighted Sums of Stretched Exponential Random Variables

Uniform Estimate for the Tail Probabilities of Randomly Weighted Sums

Computation of marginal eigenvalue distributions in the Laguerre and Jacobi $β$ ensembles

Divergence based robust estimation of the tail index through an exponential regression model

Asymptotic Gap Probability Distributions of the Gaussian Unitary Ensembles and Jacobi Unitary Ensembles

The extremal landscape for the C$β$E ensemble

Estimation of extreme risk regions under multivariate regular variation

Precise Asymptotics of Bagging Regularized M-estimators

Beta Jacobi ensembles and associated Jacobi polynomials, II

Asymptotic Tail Probability of Randomly Weighted Sum of Dependent Heavy-Tailed Random Variables