Abstract:Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In this regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibit behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and generalization errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for both deterministic and random teacher models. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.

Kernel entropy estimation for linear processes II

Kernel Generated Two-Time Parameter Gaussian Processes and Some of Their Path Properties

Random Weighting Estimation Of Kernel Density

Kernel Estimation of the Extropy Function under α-Mixing Dependent Data

Entropy Estimate for Degenerate SDEs with Applications to Nonlinear Kinetic Fokker-Planck Equations

Measures of Entropy From Data Using Infinitely Divisible Kernels

Berry–Esseen Bounds for Kernel Estimates of Stationaryprocesses

Heat kernel estimates for non-symmetric stable-like processes

Exponential inequalities under the sub-linear expectations with applications to laws of the iterated logarithm

Efficient multivariate entropy estimation via $k$-nearest neighbour distances

Heat kernel estimates for nonlocal kinetic operators

Entropy Estimate for Degenerate SDEs with Applications to Nonlinear Kinetic Fokker–Planck Equations

Probability Density Function Estimation Using Gamma Kernels

Empirical estimation of entropy functionals with confidence

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Rényi’s Entropy Perspective

Second-Moment/Order Approximations by Kernel Smoothers with Application to Volatility Estimation

Dirichlet Heat kernel estimates for a large class of anisotropic Markov processes

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective

Kernel Estimation of Density Level Sets

Universality of kernel random matrices and kernel regression in the quadratic regime

Heat kernel estimates and their stabilities for symmetric jump processes with general mixed polynomial growths on metric measure spaces