Abstract:Consider the problem: given the data pair <math>(x,y)</math> drawn from a population with <math>f*(x)=E[y|x=x]</math>, specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does ft, the function computed by the neural network at time t, relate to <math>f*</math>, in terms of approximation and representation? What are the provable benefits of the adaptive representation by neural networks compared to the prespecified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks. First, we show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation and performs the global least-squares projection onto the adaptive RKHS, simultaneously. Second, we prove that as the RKHS is data-adaptive and task-specific, the residual for <math>f*</math> lies in a subspace that is potentially much smaller than the orthogonal complement of the RKHS. The result formalizes the representation and approximation benefits of neural networks. Finally, we show that the neural network function computed by gradient flow converges to the kernel ridgeless regression with an adaptive kernel, in the limit of vanishing regularization. The adaptive kernel viewpoint provides new angles of studying the approximation, representation, generalization, and optimization advantages of neural networks. <a class="ext-link" href="https://doi.org/10.1080/01621459.2020.1745812">Supplementary materials</a> for this article are available online.

Approximation and Gradient Descent Training with Neural Networks

Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

Function Gradient Approximation with Random Shallow ReLU Networks with Control Applications

Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

Gradient-enhanced deep neural network approximations

Numerical Analysis on Neural Network Projected Schemes for Approximating One Dimensional Wasserstein Gradient Flows

Approximation and interpolation of deep neural networks

A Gradient Free Neural Network Framework Based on Universal Approximation Theorem

Accelerated CNN Training Through Gradient Approximation

On the convergence of gradient descent for two layer neural networks

Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time

How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent

Preconditioned Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression

Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization

Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

Gradient Networks

The limitation of neural nets for approximation and optimization

Convergence of continuous-time stochastic gradient descent with applications to linear deep neural networks

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks

Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks