Abstract:We obtain upper bounds for the estimation error of Kernel Ridge Regression (KRR) for all non-negative regularization parameters, offering a geometric perspective on various phenomena in KRR. As applications: 1. We address the multiple descent problem, unifying the proofs of <a class="link-https" data-arxiv-id="1908.10292" href="https://arxiv.org/abs/1908.10292">arXiv:1908.10292</a> and <a class="link-https" data-arxiv-id="1904.12191" href="https://arxiv.org/abs/1904.12191">arXiv:1904.12191</a> for polynomial kernels and we establish multiple descent for the upper bound of estimation error of KRR under sub-Gaussian design and non-asymptotic regimes. 2. For a sub-Gaussian design vector and for non-asymptotic scenario, we prove a one-sided isomorphic version of the Gaussian Equivalent Conjecture. 3. We offer a novel perspective on the linearization of kernel matrices of non-linear kernel, extending it to the power regime for polynomial kernels. 4. Our theory is applicable to data-dependent kernels, providing a convenient and accurate tool for the feature learning regime in deep learning theory. 5. Our theory extends the results in <a class="link-https" data-arxiv-id="2009.14286" href="https://arxiv.org/abs/2009.14286">arXiv:2009.14286</a> under weak moment assumption. Our proof is based on three mathematical tools developed in this paper that can be of independent interest: 1. Dvoretzky-Milman theorem for ellipsoids under (very) weak moment assumptions. 2. Restricted Isomorphic Property in Reproducing Kernel Hilbert Spaces with embedding index conditions. 3. A concentration inequality for finite-degree polynomial kernel functions.

Reduced Rank Multivariate Kernel Ridge Regression

Robust reduced-rank regression

Kernel Learning in Ridge Regression "Automatically" Yields Exact Low Rank Solution

Ridgelet Kernel Regression

A novel ridgelet kernel regression method

Low-rank kernel regression with preserved locality for multi-class analysis

Multivariate reduced rank regression by signal subspace matching

Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration

Robust Regularized Low-Rank Matrix Models for Regression and Classification

Kernel Ridge Regression Inference

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

Robust reduced rank regression in a distributed setting

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Universality of kernel random matrices and kernel regression in the quadratic regime

A framework of regularized low-rank matrix models for regression and classification

A Bound on the Maximal Marginal Degrees of Freedom

A Geometrical Analysis of Kernel Ridge Regression and its Applications

Sparse Reduced Rank Huber Regression in High Dimensions

Best Subset Selection in Reduced Rank Regression

Adaptive estimation of the rank of the coefficient matrix in high dimensional multivariate response regression models

Stochastic Low-Rank Kernel Learning for Regression