Abstract:Empirical process theory for i.i.d. observations has emerged as a ubiquitous tool for understanding the generalization properties of various statistical problems. However, in many applications where the data exhibit temporal dependencies (e.g., in finance, medical imaging, weather forecasting etc.), the corresponding empirical processes are much less understood. Motivated by this observation, we present a general bound on the expected supremum of empirical processes under standard $\beta/\rho$-mixing assumptions. Unlike most prior work, our results cover both the long and the short-range regimes of dependence. Our main result shows that a non-trivial trade-off between the complexity of the underlying function class and the dependence among the observations characterizes the learning rate in a large class of nonparametric problems. This trade-off reveals a new phenomenon, namely that even under long-range dependence, it is possible to attain the same rates as in the i.i.d. setting, provided the underlying function class is complex enough. We demonstrate the practical implications of our findings by analyzing various statistical estimators in both fixed and growing dimensions. Our main examples include a comprehensive case study of generalization error bounds in nonparametric regression over smoothness classes in fixed as well as growing dimension using neural nets, shape-restricted multivariate convex regression, estimating the optimal transport (Wasserstein) distance between two probability distributions, and classification under the Mammen-Tsybakov margin condition -- all under appropriate mixing assumptions. In the process, we also develop bounds on $L_r$ ($1\le r\le 2$)-localized empirical processes with dependent observations, which we then leverage to get faster rates for (a) tuning-free adaptation, and (b) set-structured learning problems.

Sparse-penalized deep neural networks estimator under weak dependence

Penalized deep neural networks estimator with general loss functions under weak dependence

Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality

Excess risk bound for deep learning under weak dependence

Deep learning for ψ -weakly dependent processes

Statistical learning by sparse deep neural networks

Penalized least squares estimation with weakly dependent data

Robust deep learning from weakly dependent data

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Elastic Net with Adaptive Weight for Image Denoising

NONCONCAVE PENALIZED M-ESTIMATION WITH A DIVERGING NUMBER OF PARAMETERS

Wasserstein-based Minimax Estimation of Dependence in Multivariate Regularly Varying Extremes

Penalized robust estimators in logistic regression with applications to sparse models

Statistical Properties of Deep Neural Networks with Dependent Data

Penalized Sparse Covariance Regression with High Dimensional Covariates

Trade-off Between Dependence and Complexity for Nonparametric Learning -- an Empirical Process Approach

Stability of the Elastic Net Estimator

Nonnegative Estimation and Variable Selection under Minimax Concave Penalty for Sparse High-Dimensional Linear Regression Models

Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference