Abstract:Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

Stochastic Low-Rank Kernel Learning for Regression

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Kernel Learning in Ridge Regression "Automatically" Yields Exact Low Rank Solution

Conjugate Gradients for Kernel Machines

Optimal learning rates for Kernel Conjugate Gradient regression

Low-rank kernel regression with preserved locality for multi-class analysis

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Learning the kernel matrix via predictive low-rank approximations

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Constrained recursive kernel density/regression estimation by stochastic quasi-gradient methods

Learning-Based Low-Rank Approximations

Kernel Regression with Sparse Metric Learning

Reduced Rank Multivariate Kernel Ridge Regression

Kernel-based function learning in dynamic and non stationary environments

Errors-in-variables Fréchet Regression with Low-rank Covariate Approximation

Kernel partial least squares for stationary data

Robust Regularized Low-Rank Matrix Models for Regression and Classification

Efficient kernel surrogates for neural network-based regression

Solving Kernel Ridge Regression with Gradient-Based Optimization Methods

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

Multistage Stochastic Optimization via Kernels