Abstract:What is the smallest multilayer perceptron able to compute arbitrary and random functions? Previous results show that a net with one hidden layer containing N − 1 threshold units is capable of implementing an arbitrary dichotomy of N points. A construction is presented here for implementing an arbitrary dichotomy with one hidden layer containing [Nd] units, for any set of N points in general position in d dimensions. This is in fact the smallest such net as dichotomies which cannot be implemented by any net with fewer units are described. Several constructions are presented of one-hidden-layer nets implementing arbitrary functions into the e-dimensional hypercube. One of these has only [4Nd][e[log2(Nd)]] units in its hidden layer. Arguments based on a function counting theorem of Cover establish that any net implementing arbitrary functions must have at least Nelog2(N) weights, so that no net with one hidden layer containing less than Ne/(d log2(N)) units will suffice. Simple counts also show that if the weights are only allowed to assume one of ng possible values, no net with fewer than Nelog2(ng) weights will suffice. Thus the gain coming from using real valued synapses appears to be only logarithmic. The circuit implementing functions into the e hypercube realizes such logarithmic gains. Since the counting arguments limit below only the number of weights, the possibility is suggested that, if suitable restrictions are imposed on the input vector set to avoid topological obstructions, two-hidden-layer nets with O(N) weights but only O(√N) threshold units might suffice for arbitrary dichotomies. Interesting and potentially sufficient restrictions include (a) if the vectors are binary, i.e., lie on the d hypercube or (b) if they are randomly and uniformly selected from a bounded region.

Classification Ability of Single Hidden Layer Feedforward Neural Networks

Transparent Classification with Multilayer Logical Perceptrons and Random Binarization

Multilayer neural networks with extensively many hidden units

Interpretable neural networks based on continuous-valued logic and multicriteria decision operators

Construct H-MFNN by Complexity Reduction Approach.

Nearly-tight bounds on linear regions of piecewise linear neural networks

On the approximation by single hidden layer feedforward neural networks with fixed weights

High-dimensional classification problems with Barron regular boundaries under margin conditions

The Geometric Structure of Fully-Connected ReLU Layers

On the Number of Linear Regions of Deep Neural Networks

Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations

Multilayer Dense Connections for Hierarchical Concept Classification

Analysis on the Number of Linear Regions of Piecewise Linear Neural Networks

Convergence and objective functions of noise-injected multilayer perceptrons with hidden multipliers

Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks

On the Principles of ReLU Networks with One Hidden Layer

A New Method for Decision on the Structure of RBF Neural Network

BCMLP: Binary-connected multilayer perceptrons

Dissecting Deep Neural Networks

A Significantly Better Class of Activation Functions Than ReLU Like Activation Functions

On the capabilities of multilayer perceptrons