Abstract:We develop a geometric approximation theory for deep feed-forward neural networks with ReLU activations. Given a $d$-dimensional hypersurface in $\mathbb{R}^{d+1}$ represented as the graph of a $C^2$-function $\phi$, we show that a deep fully-connected ReLU network of width $d+1$ can implicitly construct an approximation as its zero contour with a precision bound depending on the number of layers. This result is directly applicable to the binary classification setting where the sign of the network is trained as a classifier, with the network's zero contour as a decision boundary. Our proof is constructive and relies on the geometrical structure of ReLU layers provided in [doi:<a class="link-https link-external" data-doi="10.48550/arXiv.2310.03482" href="https://doi.org/10.48550/arXiv.2310.03482" rel="external noopener nofollow">https://doi.org/10.48550/arXiv.2310.03482</a>]. Inspired by this geometrical description, we define a new equivalent network architecture that is easier to interpret geometrically, where the action of each hidden layer is a projection onto a polyhedral cone derived from the layer's parameters. By repeatedly adding such layers, with parameters chosen such that we project small parts of the graph of $\phi$ from the outside in, we, in a controlled way, construct a network that implicitly approximates the graph over a ball of radius $R$. The accuracy of this construction is controlled by a discretization parameter $\delta$ and we show that the tolerance in the resulting error bound scales as $(d-1)R^{3/2}\delta^{1/2}$ and the required number of layers is of order $d\big(\frac{32R}{\delta}\big)^{\frac{d+1}{2}}$.

Two-layer Networks with the ReLU^k Activation Function: Barron Spaces and Derivative Approximation

On the Banach Spaces Associated with Multi-Layer ReLU Networks: Function Representation, Approximation Theory and Gradient Descent Dynamics

Adaptive two-layer ReLU neural network: I. Best least-squares approximation

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Neural networks with ReLU powers need less depth

Function and derivative approximation by shallow neural networks

Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

Training a Two Layer ReLU Network Analytically

How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer

Learning Two-Layer ReLU Networks Is Nearly as Easy as Learning Linear Classifiers on Separable Data

Rates of Approximation by ReLU Shallow Neural Networks

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

ReLU neural network approximation to piecewise constant functions

Optimal function approximation with ReLU neural networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

The Barron Space and the Flow-Induced Function Spaces for Neural Network Models

Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression

Approximation of functions from Korobov spaces by shallow neural networks

Nearly Optimal Approximation Rates for Deep Super ReLU Networks on Sobolev Spaces

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks