Abstract:Consider the problem: given the data pair <math>(x,y)</math> drawn from a population with <math>f*(x)=E[y|x=x]</math>, specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does ft, the function computed by the neural network at time t, relate to <math>f*</math>, in terms of approximation and representation? What are the provable benefits of the adaptive representation by neural networks compared to the prespecified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks. First, we show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation and performs the global least-squares projection onto the adaptive RKHS, simultaneously. Second, we prove that as the RKHS is data-adaptive and task-specific, the residual for <math>f*</math> lies in a subspace that is potentially much smaller than the orthogonal complement of the RKHS. The result formalizes the representation and approximation benefits of neural networks. Finally, we show that the neural network function computed by gradient flow converges to the kernel ridgeless regression with an adaptive kernel, in the limit of vanishing regularization. The adaptive kernel viewpoint provides new angles of studying the approximation, representation, generalization, and optimization advantages of neural networks. <a class="ext-link" href="https://doi.org/10.1080/01621459.2020.1745812">Supplementary materials</a> for this article are available online.

Kernel Neuron And Its Training Algorithm

Sparse Training Procedure for Kernel Neuron

A Learning Algorithm with Gaussian Regularizer for Kernel Neuron

Adaptive training of a kernel-based nonlinear discriminator

Adaptive Training of a Kernel-Based Representative and Discriminative Nonlinear Classifier

A Nonlinear Perceptron Algorithm Based on Kernel Functions

Training Radial Basis Function Classifiers with Gaussian Kernel Clustering and Fuzzy Decision Technique

Nonlinear Pairwise Layer and Its Training for Kernel Learning

Nonlinear Pocket Algorithm with Kernels

A Unified Kernel for Neural Network Learning

Kernel Generalized Nonlinear Discriminant Analysis Algorithm For Pattern Recognition

An Adaptively Trained Kernel-Based Nonlinear Representor for Handwritten Digit Classification

A Multiclass Kernel Perceptron Algorithm

Kernel Nearest-Neighbor Algorithm

Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

The Kernel Conjugate Gradient Algorithms.

An Efficient Kernel-Based Nonlinear Regression Method For Two-Class Classification

A Distributed Algorithm for Training Nonlinear Kernel Machines

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

Kernel Representation-Based Nearest Neighbor Classifier