Abstract:Kernel ridge regression (KRR) is a fundamental computational tool, appearing in problems that range from computational chemistry to health analytics, with a particular interest due to its starring role in Gaussian process regression. However, it is challenging to scale KRR solvers to large datasets: with $n$ training points, a direct solver (i.e., Cholesky decomposition) uses $O(n^2)$ storage and $O(n^3)$ flops. Iterative methods for KRR, such as preconditioned conjugate gradient (PCG), avoid the cubic scaling of direct solvers and often use low-rank preconditioners; a rank $r$ preconditioner uses $O(rn)$ storage and each iteration requires $O(n^2)$ flops. To reduce the storage and iteration complexity of iterative solvers for KRR, we propose ASkotch ($\textbf{A}$ccelerated $\textbf{s}$calable $\textbf{k}$ernel $\textbf{o}$p$\textbf{t}$imization using block $\textbf{c}$oordinate descent with $\textbf{H}$essian preconditioning). For a given block size $|b| << n$, each iteration of ASkotch uses $O(r|b| + n)$ storage and $O(n|b|)$ flops, so ASkotch scales better than Cholesky decomposition and PCG. We prove that ASkotch obtains linear convergence to the optimum, with the convergence rate depending on the square roots of the $\textit{preconditioned}$ block condition numbers. Furthermore, we solve KRR problems that were considered to be impossibly large while using limited computational resources: we show that ASkotch outperforms PCG methods with respect to generalization error on large-scale KRR (up to $n = 10^8$) and KRR classification tasks (up to $n = 10^7$) while running each of our experiments on $\textit{a single 12 GB Titan V GPU}$. Our work opens up the possibility of as-yet-unimagined applications of KRR across a wide range of disciplines.

Snacks: a fast large-scale kernel SVM solver

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity.

Recipe for Fast Large-scale SVM Training: Polishing, Parallelism, and more RAM!

Online Kernel Learning with a Near Optimal Sparsity Bound

FALKON: An Optimal Large Scale Kernel Method

Fast Algorithms for Linear and Kernel SVM+.

Sparse Least Squares Support Vector Machine for Function Estimation

Fast Parallel SVM using Data Augmentation

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines

Linear Regression-Based Efficient SVM Learning for Large-Scale Classification.

Mini-batch Quasi-Newton Optimization for Large Scale Linear Support Vector Regression

Have ASkotch: Fast Methods for Large-scale, Memory-constrained Kernel Ridge Regression

Approximate Approach to Train SVM on Very Large Data Sets

Hash-SVM: Scalable Kernel Machines for Large-Scale Visual Classification

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

Nyström Kernel Stein Discrepancy

Fast and Accurate Refined Nyström-Based Kernel SVM

NESVM: A Fast Gradient Method for Support Vector Machines

Large Scale Online Kernel Classification

Support Feature Machines