Abstract:Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

An Exact Kernel Equivalence for Finite Classification Models

Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime

Efficient kernel surrogates for neural network-based regression

An Exact Finite-dimensional Explicit Feature Map for Kernel Functions

On the Equivalence between Neural Network and Support Vector Machine

A Revision of Neural Tangent Kernel-based Approaches for Neural Networks

On Exact Computation with an Infinitely Wide Neural Net

On the Similarity between the Laplace and Neural Tangent Kernels

Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

NeuralEF: Deconstructing Kernels by Deep Neural Networks.

Fixing the NTK: From Neural Network Linearizations to Exact Convex Programs

When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?

Equivariant Neural Tangent Kernels

Scalable Neural Network Kernels

When Do Neural Networks Outperform Kernel Methods?

Universality and Optimality of Structured Deep Kernel Networks

On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains

Kernel vs. Kernel: Exploring How the Data Structure Affects Neural Collapse

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

A Unified Kernel for Neural Network Learning

Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit