Abstract:Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In this regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibit behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and generalization errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for both deterministic and random teacher models. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.

Distributed Kernel Ridge Regression with Communications.

Distributed Randomized Sketching Kernel Learning

Lepskii Principle for Distributed Kernel Ridge Regression

Effective Distributed Learning with Random Features: Improved Bounds and Algorithms

Distributed Semi-Supervised Learning with Kernel Ridge Regression

Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos

Optimal Rates for Agnostic Distributed Learning

Distributed Learning with Indefinite Kernels

Distributed Ranking with Communications: Approximation Analysis and Applications

Optimal Convergence Rates for Distributed Nystro<spacing diaeresis>m Approximation

A Communication-Efficient Distributed Algorithm for Kernel Principal Component Analysis.

A Duality Analysis of Kernel Ridge Regression in the Noiseless Regime

Error analysis of distributed least squares ranking

Distributed Online Linear Regressions

Distributed Nonparametric Regression under Communication Constraints

Universality of kernel random matrices and kernel regression in the quadratic regime

Distributed Learning With Dependent Samples

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Distributed Online Optimization via Kernel Reproduced Gradient Descent

Scalable Dual Coordinate Descent for Kernel Methods

How Many Machines Can We Use in Parallel Computing for Kernel Ridge Regression?