Abstract:Recent research shows that the dynamics of an infinitely wide neural network (NN) trained by gradient descent can be characterized by Neural Tangent Kernel (NTK) \citep{jacot2018neural}. Under the squared loss, the infinite-width NN trained by gradient descent with an infinitely small learning rate is equivalent to kernel regression with NTK \citep{arora2019exact}. However, the equivalence is only known for ridge regression currently \citep{arora2019harnessing}, while the equivalence between NN and other kernel machines (KMs), e.g. support vector machine (SVM), remains unknown. Therefore, in this work, we propose to establish the equivalence between NN and SVM, and specifically, the infinitely wide NN trained by soft margin loss and the standard soft margin SVM with NTK trained by subgradient descent. Our main theoretical results include establishing the equivalences between NNs and a broad family of $\ell_2$ regularized KMs with finite-width bounds, which cannot be handled by prior work, and showing that every finite-width NN trained by such regularized loss functions is approximately a KM. Furthermore, we demonstrate our theory can enable three practical applications, including (i) \textit{non-vacuous} generalization bound of NN via the corresponding KM; (ii) \textit{non-trivial} robustness certificate for the infinite-width NN (while existing robustness verification methods would provide vacuous bounds); (iii) intrinsically more robust infinite-width NNs than those from previous kernel regression. Our code for the experiments is available at \url{<a class="link-external link-https" href="https://github.com/leslie-CH/equiv-nn-svm" rel="external noopener nofollow">this https URL</a>}.

On the equivalence between kernel self-organising maps and self-organising mixture density networks.

A Rigorous Link Between Self-Organizing Maps and Gaussian Mixture Models

Kernel-Based Self-Organizing Map Clustering

Learning nonlinear manifolds based on mixtures of localized linear manifolds under a self-organizing framework

Kernel-Based Mixture Mapping for Image and Text Association

A Novel Kernel-Based Maximum a Posteriori Classification Method

Generalized Semisupervised Self-Optimizing Kernel Model for Quality-Related Industrial Process Monitoring.

A Kernel Spatial Complexity-Based Nonlinear Unmixing Method of Hyperspectral Imagery.

A Comparative Study of Marginalized Graph Kernel and Message-Passing Neural Network

Non-Euclidean Self-Organizing Maps

A New Adaptive Self-Organizing Map.

An Introduction to Kernel and Operator Learning Methods for Homogenization by Self-consistent Clustering Analysis

Kernel-Based Mixture Of Experts Models For Linear Regression

Self organising maps for visualising and modelling

Mixture Correntropy-Based Kernel Extreme Learning Machines

Optimal Transport for Kernel Gaussian Mixture Models

SMLSOM: The shrinking maximum likelihood self-organizing map

Self-Organizing Mixture Networks for Representation of Grayscale Digital Images

An Exact Kernel Equivalence for Finite Classification Models

Machine learning embedded EM algorithms for semiparametric mixture regression models

On the Equivalence between Neural Network and Support Vector Machine