Abstract:Kernel ridge regression (KRR) is a fundamental computational tool, appearing in problems that range from computational chemistry to health analytics, with a particular interest due to its starring role in Gaussian process regression. However, it is challenging to scale KRR solvers to large datasets: with $n$ training points, a direct solver (i.e., Cholesky decomposition) uses $O(n^2)$ storage and $O(n^3)$ flops. Iterative methods for KRR, such as preconditioned conjugate gradient (PCG), avoid the cubic scaling of direct solvers and often use low-rank preconditioners; a rank $r$ preconditioner uses $O(rn)$ storage and each iteration requires $O(n^2)$ flops. To reduce the storage and iteration complexity of iterative solvers for KRR, we propose ASkotch ($\textbf{A}$ccelerated $\textbf{s}$calable $\textbf{k}$ernel $\textbf{o}$p$\textbf{t}$imization using block $\textbf{c}$oordinate descent with $\textbf{H}$essian preconditioning). For a given block size $|b| << n$, each iteration of ASkotch uses $O(r|b| + n)$ storage and $O(n|b|)$ flops, so ASkotch scales better than Cholesky decomposition and PCG. We prove that ASkotch obtains linear convergence to the optimum, with the convergence rate depending on the square roots of the $\textit{preconditioned}$ block condition numbers. Furthermore, we solve KRR problems that were considered to be impossibly large while using limited computational resources: we show that ASkotch outperforms PCG methods with respect to generalization error on large-scale KRR (up to $n = 10^8$) and KRR classification tasks (up to $n = 10^7$) while running each of our experiments on $\textit{a single 12 GB Titan V GPU}$. Our work opens up the possibility of as-yet-unimagined applications of KRR across a wide range of disciplines.

Embrace rejection: Kernel matrix approximation by accelerated randomly pivoted Cholesky

Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluations

Kernel quadrature with randomly pivoted Cholesky

Randomly Pivoted Partial Cholesky: Random How?

CholeskyQR with Randomization and Pivoting for Tall Matrices (CQRRPT)

Robust, randomized preconditioning for kernel ridge regression

PBBFMM3D: A parallel black-box algorithm for kernel matrix-vector multiplication

Data-Driven Linear Complexity Low-Rank Approximation of General Kernel Matrices: A Geometric Approach

Low-Rank Kernel Matrix Approximation Using Skeletonized Interpolation With Endo- or Exo-Vertices

Approximate Gaussian Elimination for Laplacians: Fast, Sparse, and Simple

On the Unreasonable Effectiveness of Single Vector Krylov Methods for Low-Rank Approximation

Efficient randomized tensor-based algorithms for function approximation and low-rank kernel interactions

Efficient Randomized Algorithms for the Fixed-Precision Low-Rank Matrix Approximation

Have ASkotch: Fast Methods for Large-scale, Memory-constrained Kernel Ridge Regression

A randomized preconditioned Cholesky-QR algorithm

Adaptive Parallelizable Algorithms for Interpolative Decompositions via Partially Pivoted LU

Efficient error and variance estimation for randomized matrix computations

Subspace embedding with random Khatri-Rao products and its application to eigensolvers

Large-scale Nyström kernel matrix approximation using randomized SVD.

Sketched and truncated polynomial Krylov methods: Evaluation of matrix functions

Analysis of Randomized Householder-Cholesky QR Factorization with Multisketching