Abstract:We obtain upper bounds for the estimation error of Kernel Ridge Regression (KRR) for all non-negative regularization parameters, offering a geometric perspective on various phenomena in KRR. As applications: 1. We address the multiple descent problem, unifying the proofs of <a class="link-https" data-arxiv-id="1908.10292" href="https://arxiv.org/abs/1908.10292">arXiv:1908.10292</a> and <a class="link-https" data-arxiv-id="1904.12191" href="https://arxiv.org/abs/1904.12191">arXiv:1904.12191</a> for polynomial kernels and we establish multiple descent for the upper bound of estimation error of KRR under sub-Gaussian design and non-asymptotic regimes. 2. For a sub-Gaussian design vector and for non-asymptotic scenario, we prove a one-sided isomorphic version of the Gaussian Equivalent Conjecture. 3. We offer a novel perspective on the linearization of kernel matrices of non-linear kernel, extending it to the power regime for polynomial kernels. 4. Our theory is applicable to data-dependent kernels, providing a convenient and accurate tool for the feature learning regime in deep learning theory. 5. Our theory extends the results in <a class="link-https" data-arxiv-id="2009.14286" href="https://arxiv.org/abs/2009.14286">arXiv:2009.14286</a> under weak moment assumption. Our proof is based on three mathematical tools developed in this paper that can be of independent interest: 1. Dvoretzky-Milman theorem for ellipsoids under (very) weak moment assumptions. 2. Restricted Isomorphic Property in Reproducing Kernel Hilbert Spaces with embedding index conditions. 3. A concentration inequality for finite-degree polynomial kernel functions.

A Duality Analysis of Kernel Ridge Regression in the Noiseless Regime

Universality of kernel random matrices and kernel regression in the quadratic regime

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

A Geometrical Analysis of Kernel Ridge Regression and its Applications

Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime

A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator

Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration

Kernel Ridge Regression for General Noise Model with Its Application.

Robust Regularized Kernel Regression.

Low-rank kernel regression with preserved locality for multi-class analysis

Asymptotic Theory for Linear Functionals of Kernel Ridge Regression

Solving Kernel Ridge Regression with Gradient Descent for a Non-Constant Kernel

Kernel Ridge Regression Inference

On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay

Solving Kernel Ridge Regression with Gradient-Based Optimization Methods

Distributed Kernel Ridge Regression with Communications.

Ridgeless Regression with Random Features

On the Optimality of Misspecified Kernel Ridge Regression

Kernel Learning in Ridge Regression "Automatically" Yields Exact Low Rank Solution