Abstract:Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

SLiKER: Sparse Loss Induced Kernel Ensemble Regression

Dynamically Weighted Ensemble Neural Networks for Regression Problems

Robust kernel ensemble regression in diversified kernel space with shared parameters

Online Kernel Learning with a Near Optimal Sparsity Bound

Diversified deep hierarchical kernel ensemble regression

Kernel Regression with Sparse Metric Learning

A hybrid ensemble method with negative correlation learning for regression

Designing Ensemble Learning Algorithms Using Kernel Methods

Cocktail Ensemble for Regression

Robust Sparse Kernel Density Estimation by Inducing Randomness

Linear Programming-Based Sparse Kernel Regression with L1-Norm Minimization for Nonlinear System Modeling

ERDeR: The Combination of Statistical Shrinkage Methods and Ensemble Approaches to Improve the Performance of Deep Regression

Sparse Training Procedure for Kernel Neuron

Non-uniform Nyström approximation for sparse kernel regression: Theoretical analysis and experimental evaluation

Ensemble Sparse Models for Image Analysis

Sparse Multiple Kernel Learning: Minimax Rates with Random Projection

Sparse kernel deep stacking networks

Efficient kernel surrogates for neural network-based regression

An interpretable regression approach based on bi-sparse optimization

Convergence of uncertainty estimates in Ensemble and Bayesian sparse model discovery

Sparse neural network regression with variable selection