Abstract:Empirical process theory for i.i.d. observations has emerged as a ubiquitous tool for understanding the generalization properties of various statistical problems. However, in many applications where the data exhibit temporal dependencies (e.g., in finance, medical imaging, weather forecasting etc.), the corresponding empirical processes are much less understood. Motivated by this observation, we present a general bound on the expected supremum of empirical processes under standard $\beta/\rho$-mixing assumptions. Unlike most prior work, our results cover both the long and the short-range regimes of dependence. Our main result shows that a non-trivial trade-off between the complexity of the underlying function class and the dependence among the observations characterizes the learning rate in a large class of nonparametric problems. This trade-off reveals a new phenomenon, namely that even under long-range dependence, it is possible to attain the same rates as in the i.i.d. setting, provided the underlying function class is complex enough. We demonstrate the practical implications of our findings by analyzing various statistical estimators in both fixed and growing dimensions. Our main examples include a comprehensive case study of generalization error bounds in nonparametric regression over smoothness classes in fixed as well as growing dimension using neural nets, shape-restricted multivariate convex regression, estimating the optimal transport (Wasserstein) distance between two probability distributions, and classification under the Mammen-Tsybakov margin condition -- all under appropriate mixing assumptions. In the process, we also develop bounds on $L_r$ ($1\le r\le 2$)-localized empirical processes with dependent observations, which we then leverage to get faster rates for (a) tuning-free adaptation, and (b) set-structured learning problems.

Maximal Inequalities for Empirical Processes under General Mixing Conditions with an Application to Strong Approximations

Strong Approximation for A Class of Stationary Processes

On Bernstein Type Inequalities for Stochastic Integrals of Multivariate Point Processes

Strong Approximation for Ρ-Mixing Sequences

Limit theorems for kernel density estimators under dependent samples

Concentration inequalities for locally small increments of compound empirical processes with applications to solutions of compound and risk averse stochastical programming

Generalization for slowly mixing processes

Strong Approximations for Empirical Processes Indexed by Lipschitz Functions

Strong Approximation Theorems for Sums of Random Variables when Extreme Terms Are Excluded

Asymptotic Properties of Nonparametric M-Estimation for Mixing Functional Data

Bernstein inequality and moderate deviations under strong mixing conditions

On the rate of convergence in the central limit theorem for random sums of strongly mixing random variables

Strongly convergent homogeneous approximations to inhomogeneous Markov jump processes and applications

Mixing it up: A general framework for Markovian statistics

Mixing properties of nonstationary multivariate count processes

On the Concentration of the Minimizers of Empirical Risks

Probabilistic limit behaviors of numerical discretizations for time-homogeneous Markov processes

Trade-off Between Dependence and Complexity for Nonparametric Learning -- an Empirical Process Approach

Some strong limit theorems in averaging

Exact Exponents for Concentration and Isoperimetry in Product Polish Spaces

On the Weak Convergence of the Function-Indexed Sequential Empirical Process and its Smoothed Analogue under Nonstationarity