Abstract:Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In this regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibit behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and generalization errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for both deterministic and random teacher models. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.

Kernel Matrix Learning with a General Regularized Risk Functional Criterion

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Self-representative kernel concept factorization

Robust Regularized Low-Rank Matrix Models for Regression and Classification

Universality of kernel random matrices and kernel regression in the quadratic regime

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Low-rank kernel regression with preserved locality for multi-class analysis

A Learning Algorithm with Gaussian Regularizer for Kernel Neuron

Kernel Regression with Sparse Metric Learning

Indefinite Kernel Logistic Regression with Concave-Inexact-Convex Procedure

Learning Latent Features With Pairwise Penalties in Low-Rank Matrix Completion

Matrix-Regularized Multiple Kernel Learning Via &Lt;inline-Formula> &Lt;tex-Math Notation="latex">$(r,~p)$ &Lt;/tex-Math> &Lt;/inline-Formula> Norms

Optimal Kernel Quantile Learning with Random Features

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Theoretical Analysis of Divide-and-Conquer ERM: Beyond Square Loss and RKHS.

Learning the kernel matrix by resampling.

Kernelized Heterogeneous Risk Minimization

Generalization Properties of Hyper-Rkhs and Its Applications

A High-Order Norm-Product Regularized Multiple Kernel Learning Framework for Kernel Optimization

Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators

Generalized Nonconvex Nonsmooth Low-Rank Matrix Recovery Framework with Feasible Algorithm Designs and Convergence Analysis.