Abstract:Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

KKT-Informed Neural Network

Karush-Kuhn-Tucker Condition-Trained Neural Networks (KKT Nets)

Neural Network Solution for General Nonlinear Optimization Problems

A Recurrent Neural Network for Solving Nonconvex Optimization Problems

A discrete-time neural network for optimization problems with hybrid constraints.

An effective algorithm for hyperparameter optimization of neural networks

Convergence of a Recurrent Neural Network for Nonconvex Optimization Based on an Augmented Lagrangian Function

Optimization-Informed Neural Networks

Efficient kernel surrogates for neural network-based regression

Teaching Networks to Solve Optimization Problems

Neural Network Approaches for Parameterized Optimal Control

GPINN with Neural Tangent Kernel Technique for Nonlinear Two Point Boundary Value Problems

Self-Supervised Learning of Iterative Solvers for Constrained Optimization

Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data

A Gradient-Based Neural Network Method for Solving Strictly Convex Quadratic Programming Problems

A Revision of Neural Tangent Kernel-based Approaches for Neural Networks

A Capable Neural Network Framework for Solving Degenerate Quadratic Optimization Problems with an Application in Image Fusion

Variational Inference for Nonlinear Inverse Problems via Neural Net Kernels: Comparison to Bayesian Neural Networks, Application to Topology Optimization

Regularized Gauss-Newton for Optimizing Overparameterized Neural Networks

A new recurrent neural network for solving convex quadratic programming problems with an application to the κ-winners-take-all problem

Scalable Neural Network Kernels