Abstract:Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of $\varepsilon$ uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on $\varepsilon$) require $\Omega(\text{poly}(1/\varepsilon))$ neurons while deep networks (i.e., networks whose depth grows with $1/\varepsilon$) require $\mathcal{O}(\text{polylog}(1/\varepsilon))$ neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular type of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.

Function Approximation by Neural Networks

Fuzzy RBF Neural Network Based Function Approximation

An approximation by neural networkswith a fixed weight

Investigation of Neural Networks for Function Approximation

Approximation Capability to Functions of Several Variables, Nonlinear Functionals and Operators by Radial Basis Function Neural Networks

Approximation of RKHS Functionals by Neural Networks

Approximation to a Compact Set of Functions by Feedforward Neural Networks.

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

A Novel Radial Basis Function Neural Network For Approximation

Almost optimal estimates for approximation and learning by radial basis function networks

Function Approximation by Random Neural Networks with a Bounded Number of Layers

Approximation Capability of Interpolation Neural Networks

Frequency Domain Analysis Based Rbf Networks And Their Applications To Function Approximations

Universal Approximation Using Radial-Basis-Function Networks

A RBF Neural Network Based on Fourier Transformation for Function Approximation

L P Approximation Capability of RBF Neural Networks

Why Deep Neural Networks for Function Approximation?

APPROXIMATION MULTIDIMENSION FUCTION WITH FUNCTIONAL NETWORK

Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications

On the universal approximation property of radial basis function neural networks