Abstract:Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In this regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibit behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and generalization errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for both deterministic and random teacher models. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.

Asymptotic Distribution for Regression in A Symmetric Periodic Gaussian Kernel Hilbert Space

ASYMPTOTIC NORMALITY OF KERNEL ESTIMATES OF A DENSITY FUNCTION UNDER ASSOCIATION DEPENDENCE

Asymptotic Distributions of Innovation Density Estimators in Linear Processes

Asymptotic Distributions of Nonparametric Regression Estimators for Longitudinal or Functional Data

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Generalization Properties of Hyper-Rkhs and Its Applications

Coefficient-based regularized distribution regression

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Convergence Conditions of Online Regularized Statistical Learning in Reproducing Kernel Hilbert Space With Non-Stationary Data

Asymptotic Theory for Linear Functionals of Kernel Ridge Regression

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Asymptotic of the Lr-norm of density estimators in the autoregressive time series

Online Learning in Reproducing Kernel Hilbert Space with Non-IID Data

Universality of kernel random matrices and kernel regression in the quadratic regime

Statistical inference in the partial linear models with the inverse gaussian kernel

Asymptotics of Linear Regression with Linearly Dependent Data

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Adaptive Bayesian Regression on Data with Low Intrinsic Dimensionality

Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions

Functional regression with dependent error and missing observation in reproducing kernel Hilbert spaces