Abstract:Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

Gradient Kernel Regression

Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels

How does a kernel based on gradients of infinite-width neural networks come to be widely used: a review of the neural tangent kernel

A generalized neural tangent kernel for surrogate gradient learning

Efficient kernel surrogates for neural network-based regression

Reverse Engineering the Neural Tangent Kernel

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

On the Similarity between the Laplace and Neural Tangent Kernels

Equivariant Neural Tangent Kernels

Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

A Revision of Neural Tangent Kernel-based Approaches for Neural Networks

Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

GPINN with Neural Tangent Kernel Technique for Nonlinear Two Point Boundary Value Problems

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime

How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networks

When Do Neural Networks Outperform Kernel Methods?

Neural Tangent Kernels Motivate Graph Neural Networks with Cross-Covariance Graphs

Solving Kernel Ridge Regression with Gradient Descent for a Non-Constant Kernel

Kernel Neuron And Its Training Algorithm

When and why PINNs fail to train: A neural tangent kernel perspective