Abstract:What is the smallest multilayer perceptron able to compute arbitrary and random functions? Previous results show that a net with one hidden layer containing N − 1 threshold units is capable of implementing an arbitrary dichotomy of N points. A construction is presented here for implementing an arbitrary dichotomy with one hidden layer containing [Nd] units, for any set of N points in general position in d dimensions. This is in fact the smallest such net as dichotomies which cannot be implemented by any net with fewer units are described. Several constructions are presented of one-hidden-layer nets implementing arbitrary functions into the e-dimensional hypercube. One of these has only [4Nd][e[log2(Nd)]] units in its hidden layer. Arguments based on a function counting theorem of Cover establish that any net implementing arbitrary functions must have at least Nelog2(N) weights, so that no net with one hidden layer containing less than Ne/(d log2(N)) units will suffice. Simple counts also show that if the weights are only allowed to assume one of ng possible values, no net with fewer than Nelog2(ng) weights will suffice. Thus the gain coming from using real valued synapses appears to be only logarithmic. The circuit implementing functions into the e hypercube realizes such logarithmic gains. Since the counting arguments limit below only the number of weights, the possibility is suggested that, if suitable restrictions are imposed on the input vector set to avoid topological obstructions, two-hidden-layer nets with O(N) weights but only O(√N) threshold units might suffice for arbitrary dichotomies. Interesting and potentially sufficient restrictions include (a) if the vectors are binary, i.e., lie on the d hypercube or (b) if they are randomly and uniformly selected from a bounded region.

Multilayer neural networks with extensively many hidden units

Optimal errors and phase transitions in high-dimensional generalized linear models

Weight decay induced phase transitions in multilayer neural networks

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Multilayer In-Place Learning Networks for Modeling Functional Layers in the Laminar Cortex.

On the capabilities of multilayer perceptrons

Breaking the Curse of Dimensionality with Convex Neural Networks

Memory capacity of two layer neural networks with smooth activations

Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Isomorphisms in Multilayer Networks

Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space

On the Transformation Mechanisms of Multilayer Perceptrons with Sigmoid Activation Functions for Classifications

Multilayer In-Place Learning Networks: Multitask Invariance And Adaptive Lateral Connections

Interpretable neural networks based on continuous-valued logic and multicriteria decision operators

Layer aggregation and reducibility of multilayer interconnected networks

On the equivalence of Hopfield networks and Boltzmann Machines

Network Inference with Hidden Units

Consideration on the learning efficiency of multiple-layered neural networks with linear units

Classification Ability of Single Hidden Layer Feedforward Neural Networks

DNNs as Layers of Cooperating Classifiers