Abstract:Neural networks are versatile tools for computation, having the ability to approximate a broad range of functions. An important problem in the theory of deep neural networks is expressivity; that is, we want to understand the functions that are computable by a given network. We study real infinitely differentiable (smooth) hierarchical functions implemented by feedforward neural networks via composing simpler functions in two cases: 1) each constituent function of the composition has fewer inputs than the resulting function; 2) constituent functions are in the more specific yet prevalent form of a non-linear univariate function (e.g. tanh) applied to a linear multivariate function. We establish that in each of these regimes there exist non-trivial algebraic partial differential equations (PDEs), which are satisfied by the computed functions. These PDEs are purely in terms of the partial derivatives and are dependent only on the topology of the network. For compositions of polynomial functions, the algebraic PDEs yield non-trivial equations (of degrees dependent only on the architecture) in the ambient polynomial space that are satisfied on the associated functional varieties. Conversely, we conjecture that such PDE constraints, once accompanied by appropriate non-singularity conditions and perhaps certain inequalities involving partial derivatives, guarantee that the smooth function under consideration can be represented by the network. The conjecture is verified in numerous examples including the case of tree architectures which are of neuroscientific interest. Our approach is a step toward formulating an algebraic description of functional spaces associated with specific neural networks, and may provide new, useful tools for constructing neural networks.

A Neural Network of Smooth Hinge Functions.

Toward Deep Adaptive Hinging Hyperplanes

Generalization of hinging hyperplanes

Adaptive Hinging Hyperplane and Its Comparison with High-Level Canonical Piecewise Linear Representation

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Efficient Hinging Hyperplanes Neural Network and Its Application in Nonlinear System Identification

Adaptive Hinging Hyperplanes And Its Applications In Dynamic System Identification

Improving Algorithm of Hinging Hyperplanes for Function Approximation

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

A discrete-time neural network for optimization problems with hybrid constraints.

Fast Adaptive Hinging Hyperplanes

Learning with smooth Hinge losses

A Novel Nonlinear Approximation Approach- -Enhanced Hinge-Finding Algorithm

A special kind of neural networks: continuous piecewise linear functions

Approximating smooth functions by deep neural networks with sigmoid activation function

Neural Network for Nonsmooth, Nonconvex Constrained Minimization Via Smooth Approximation.

PDE constraints on smooth hierarchical functions computed by neural networks

A Neural Network For Smooth Nonconvex Saddle Point Problems

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

An Improved Coordinate Update Method For The Identification Of Adaptive Hinging Hyperplanes Model

Approximation by approximate interpolation neural networks with single hidden layer