Abstract:The consistency of a learning method is usually established under the assumption that the observations are a realization of an independent and identically distributed (i.i.d.) or mixing process. Yet, kernel methods such as support vector machines (SVMs), Gaussian processes, or conditional kernel mean embeddings (CKMEs) all give excellent performance under sampling schemes that are obviously non-i.i.d., such as when data comes from a dynamical system. We propose the new notion of empirical weak convergence (EWC) as a general assumption explaining such phenomena for kernel methods. It assumes the existence of a random asymptotic data distribution and is a strict weakening of previous assumptions in the field. Our main results then establish consistency of SVMs, kernel mean embeddings, and general Hilbert-space valued empirical expectations with EWC data. Our analysis holds for both finite- and infinite-dimensional outputs, as we extend classical results of statistical learning to the latter case. In particular, it is also applicable to CKMEs. Overall, our results open new classes of processes to statistical learning and can serve as a foundation for a theory of learning beyond i.i.d. and mixing.

Consistency for the Kernel-type Density Estimator in the Case of M-Dependent Samples

Limit theorems for kernel density estimators under dependent samples

Consistency of Kernel Density Estimators for Causal Processes

ASYMPTOTIC NORMALITY OF KERNEL ESTIMATES OF A DENSITY FUNCTION UNDER ASSOCIATION DEPENDENCE

M-Dependence Approximation for Dependent Random Variables

Rates of strong uniform consistency for the $k$-nearest neighbors kernel estimators of density and regression function

Strong Consistency of the Internal Estimator of Nonparametric Regression with Dependent Data

A Supplement to the Complete Convergence

Nonparametric M-estimation for Functional Stationary Ergodic Data

Asymptotic Distributions of Innovation Density Estimators in Linear Processes

Precise large deviations for randomly weighted sums of negatively dependent random variables with consistently varying tails

On the Consistency of Kernel Methods with Dependent Observations

Asymptotic Properties of Nonparametric M-Estimation for Mixing Functional Data

Kernel Estimation of Density Level Sets

The consistency of the estimators in semiparametric regression model based on m-asymptotic negatively associated errors

N-Consistent Density Estimation in Semiparametric Regression Models

On consistency for wavelet estimator of regression function based on biased samples under extended negatively dependence

Pointwise Sharp Moderate Deviations for a Kernel Density Estimator

A note on the convergence rate of the kernel density estimator of the mode

Asymptotic properties of kernel density and hazard rate function estimators with censored widely orthant dependent data

Consistency of Change Point Estimators for Symmetrical Stable Distribution with Parameters Shift