Abstract:Deep neural networks (DNNs) have emerged as a popular mathematical tool for function approximation due to their capability of modelling highly nonlinear functions. Their applications range from image classification and natural language processing to learning-based control. Despite their empirical successes, there is still a lack of theoretical understanding of the representative power of such deep architectures. In this work, we provide a theoretical analysis of the expressiveness of fully-connected, feedforward DNNs with 1-Lipschitz activation functions. In particular, we characterize the expressiveness of a DNN by its Lipchitz constant. By leveraging random matrix theory, we show that, given sufficiently large and randomly distributed weights, the expected upper and lower bounds of the Lipschitz constant of a DNN and hence their expressiveness increase exponentially with depth and polynomially with width, which gives rise to the benefit of the depth of DNN architectures for efficient function approximation. This observation is consistent with established results based on alternative expressiveness measures of DNNs. In contrast to most of the existing work, our analysis based on the Lipschitz properties of DNNs is applicable to a wider range of activation nonlinearities and potentially allows us to make sensible comparisons between the complexity of a DNN and the function to be approximated by the DNN. We consider this work to be a step towards understanding the expressive power of DNNs and towards designing appropriate deep architectures for practical applications such as system control.

The Expressive Power of Neural Networks: A View from the Width

On the Expressive Power of Neural Networks

Quasi-Equivalence of Width and Depth of Neural Networks

Neural networks with ReLU powers need less depth

Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

New advances in universal approximation with neural networks of minimal width

Towards Lower Bounds on the Depth of ReLU Neural Networks

On the Expressive Power of Deep Neural Networks

On Minimal Depth in Neural Networks

An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants

A Theoretical Study of Neural Network Expressive Power via Manifold Topology

Generalization and Expressivity for Deep Nets

Expressive Power of ReLU and Step Networks under Floating-Point Operations

Universal approximation with complex-valued deep narrow neural networks

1-WL Expressiveness Is (Almost) All You Need

Minimal Width for Universal Property of Deep RNN

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

Topological Expressivity of ReLU Neural Networks

Minimum width for universal approximation using ReLU networks on compact domain

The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks

The Expressivity and Training of Deep Neural Networks: Toward the Edge of Chaos?