Abstract:Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant quadratic neuron enjoys an enhanced expressive capability over the conventional neuron. However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been studied up to now, preferably in relation to that of a conventional neural network. Specifically, we ask four basic questions in this paper: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight into universal approximation? (4) To approximate the same class of functions with the same error bound, could a quantized quadratic network have a lower number of weights than a quantized conventional network? Our main contributions are the four interconnected theorems shedding light upon these four questions and demonstrating the merits of a quadratic network in terms of expressive efficiency, unique capability, compact architecture and computational capacity respectively. (C) 2020 Elsevier Ltd. All rights reserved.

Quasi-Equivalence of Width and Depth of Neural Networks

The Expressive Power of Neural Networks: A View from the Width

Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

Neural networks with ReLU powers need less depth

Universal approximation with quadratic deep networks

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization

Functional vs. parametric equivalence of ReLU networks

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

Generalization Ability of Wide Neural Networks on $\mathbb{R}$

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models

Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks

New advances in universal approximation with neural networks of minimal width

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Depth Separation in Norm-Bounded Infinite-Width Neural Networks

On the Equivalence Between Implicit and Explicit Neural Networks: A High-dimensional Viewpoint

On the Convergence of Deep Networks with Sample Quadratic Overparameterization

Universal Consistency of Deep ReLU Neural Networks

Minimal Width for Universal Property of Deep RNN

Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study