Abstract:The training of artificial neural networks (ANNs) is nowadays a highly relevant algorithmic procedure with many applications in science and industry. Roughly speaking, ANNs can be regarded as iterated compositions between affine linear functions and certain fixed nonlinear functions, which are usually multidimensional versions of a one-dimensional so-called activation function. The most popular choice of such a one-dimensional activation function is the rectified linear unit (ReLU) activation function which maps a real number to its positive part $ \mathbb{R} \ni x \mapsto \max\{ x, 0 \} \in \mathbb{R} $. In this article we propose and analyze a modified variant of the standard training procedure of such ReLU ANNs in the sense that we propose to restrict the negative gradient flow dynamics to a large submanifold of the ANN parameter space, which is a strict $ C^{ \infty } $-submanifold of the entire ANN parameter space that seems to enjoy better regularity properties than the entire ANN parameter space but which is also sufficiently large and sufficiently high dimensional so that it can represent all ANN realization functions that can be represented through the entire ANN parameter space. In the special situation of shallow ANNs with just one-dimensional ANN layers we also prove for every Lipschitz continuous target function that every gradient flow trajectory on this large submanifold of the ANN parameter space is globally bounded. For the standard gradient flow on the entire ANN parameter space with Lipschitz continuous target functions it remains an open problem of research to prove or disprove the global boundedness of gradient flow trajectories even in the situation of shallow ANNs with just one-dimensional ANN layers.

Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics

ReLUs Are Sufficient for Learning Implicit Neural Representations

A comparison of deep networks with ReLU activation function and linear spline-type methods

Equidistribution-based training of Free Knot Splines and ReLU Neural Networks

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Normalized gradient flow optimization in the training of ReLU artificial neural networks

On the Geometry of Deep Learning

The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks

From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training

Topological obstruction to the training of shallow ReLU neural networks

Learning Smooth Neural Functions via Lipschitz Regularization

Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification

Nonparametric regression using deep neural networks with ReLU activation function

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization

Smooth activations and reproducibility in deep networks

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries

Continuous Dynamic-NeRF: Spline-NeRF