Abstract:Deep neural networks have a good success record and are thus viewed as the best architecture choice for complex applications. Their main shortcoming has been, for a long time, the vanishing gradient which prevented the numerical optimization algorithms from acceptable convergence. A breakthrough has been achieved by the concept of residual connections -- an identity mapping parallel to a conventional layer. This concept is applicable to stacks of layers of the same dimension and substantially alleviates the vanishing gradient problem. A stack of residual connection layers can be expressed as an expansion of terms similar to the Taylor expansion. This expansion suggests the possibility of truncating the higher-order terms and receiving an architecture consisting of a single broad layer composed of all initially stacked layers in parallel. In other words, a sequential deep architecture is substituted by a parallel shallow one. Prompted by this theory, we investigated the performance capabilities of the parallel architecture in comparison to the sequential one. The computer vision datasets MNIST and CIFAR10 were used to train both architectures for a total of 6912 combinations of varying numbers of convolutional layers, numbers of filters, kernel sizes, and other meta parameters. Our findings demonstrate a surprising equivalence between the deep (sequential) and shallow (parallel) architectures. Both layouts produced similar results in terms of training and validation set loss. This discovery implies that a wide, shallow architecture can potentially replace a deep network without sacrificing performance. Such substitution has the potential to simplify network architectures, improve optimization efficiency, and accelerate the training process.

Any Deep ReLU Network is Shallow

Neural networks with ReLU powers need less depth

From Deep to Shallow: Transformations of Deep Rectifier Networks

Shallow ReLU neural networks and finite elements

Deep Representation with ReLU Neural Networks

On the Principles of ReLU Networks with One Hidden Layer

Deep ReLU Networks Have Surprisingly Simple Polytopes

Universal Consistency of Deep ReLU Neural Networks

Deep ReLU networks -- injectivity capacity upper bounds

Construction of Deep ReLU Nets for Spatially Sparse Learning

Functional vs. parametric equivalence of ReLU networks

Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification

Towards Lower Bounds on the Depth of ReLU Neural Networks

An Embedding of ReLU Networks and an Analysis of their Identifiability

Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference

Make Deep Networks Shallow Again

How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer

Stably unactivated neurons in ReLU neural networks

Most Activation Functions Can Win the Lottery Without Excessive Depth

Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training