Abstract:Empirical process theory for i.i.d. observations has emerged as a ubiquitous tool for understanding the generalization properties of various statistical problems. However, in many applications where the data exhibit temporal dependencies (e.g., in finance, medical imaging, weather forecasting etc.), the corresponding empirical processes are much less understood. Motivated by this observation, we present a general bound on the expected supremum of empirical processes under standard $\beta/\rho$-mixing assumptions. Unlike most prior work, our results cover both the long and the short-range regimes of dependence. Our main result shows that a non-trivial trade-off between the complexity of the underlying function class and the dependence among the observations characterizes the learning rate in a large class of nonparametric problems. This trade-off reveals a new phenomenon, namely that even under long-range dependence, it is possible to attain the same rates as in the i.i.d. setting, provided the underlying function class is complex enough. We demonstrate the practical implications of our findings by analyzing various statistical estimators in both fixed and growing dimensions. Our main examples include a comprehensive case study of generalization error bounds in nonparametric regression over smoothness classes in fixed as well as growing dimension using neural nets, shape-restricted multivariate convex regression, estimating the optimal transport (Wasserstein) distance between two probability distributions, and classification under the Mammen-Tsybakov margin condition -- all under appropriate mixing assumptions. In the process, we also develop bounds on $L_r$ ($1\le r\le 2$)-localized empirical processes with dependent observations, which we then leverage to get faster rates for (a) tuning-free adaptation, and (b) set-structured learning problems.

Generalization bounds for nonparametric regression with $β-$mixing samples

Learning performance of Tikhonov regularization algorithm with geometrically beta-mixing observations☆

Limit theorems for kernel density estimators under dependent samples

Estimating the Mixing Coefficients of Geometrically Ergodic Markov Processes

Sharper Bounds for Uniformly Stable Algorithms with Stationary Mixing Process

Trade-off Between Dependence and Complexity for Nonparametric Learning -- an Empirical Process Approach

Inferring the mixing properties of an ergodic process

Mixing it up: A general framework for Markovian statistics

The Generalization Performance of ERM Algorithm with Strongly Mixing Observations

Generalization bounds for mixing processes via delayed online-to-PAC conversions

Generalization Performance of Least-Square Regularized Regression Algorithm with Markov Chain Samples

Bernstein inequality and moderate deviations under strong mixing conditions

Sharp Rates in Dependent Learning Theory: Avoiding Sample Size Deflation for the Square Loss

Learning Rates of Empirical Risk Minimization Regression with Beta-Mixing Inputs

LIL Behavior for B-valued Strong Mixing Random Variables

Generalization Bounds of ERM Algorithm with Markov Chain Samples

Generalization Bounds for Dependent Data using Online-to-Batch Conversion

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Generalization for slowly mixing processes

Wasserstein-p Bounds via Cumulant-Based Edgeworth Expansion for $α$-Mixing Random Fields

Nonparametric estimation of the mixing density using polynomials