Abstract:The stunning empirical successes of neural networks currently lack rigorous theoretical explanation. What form would such an explanation take, in the face of existing complexity-theoretic lower bounds? A first step might be to show that data generated by neural networks with a single hidden layer, smooth activation functions and benign input distributions can be learned efficiently. We demonstrate here a comprehensive lower bound ruling out this possibility: for a wide class of activation functions (including all currently used), and inputs drawn from any logconcave distribution, there is a family of one-hidden-layer functions whose output is a sum gate, that are hard to learn in a precise sense: any statistical query algorithm (which includes all known variants of stochastic gradient descent with any loss function) needs an exponential number of queries even using tolerance inversely proportional to the input dimensionality. Moreover, this hard family of functions is realizable with a small (sublinear in dimension) number of activation units in the single hidden layer. The lower bound is also robust to small perturbations of the true weights. Systematic experiments illustrate a phase transition in the training error as predicted by the analysis.

Upper and lower bounds for the Lipschitz constant of random neural networks

Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

Computable Lipschitz Bounds for Deep Neural Networks

On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector

Towards Lower Bounds on the Depth of ReLU Neural Networks

Error bounds for approximations with deep ReLU neural networks in $W^{s,p}$ norms

Limitations of the Lipschitz constant as a defense against adversarial examples

Three Quantization Regimes for ReLU Networks

Adversarial Robustness of Neural Networks From the Perspective of Lipschitz Calculus: A Survey

Local Lipschitz Constant Computation of ReLU-FNNs: Upper Bound Computation with Exactness Verification

Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopes

On the Lipschitz Constant of Deep Networks and Double Descent

Risk Bounds for Robust Deep Learning

The Lipschitz Constant of Self-Attention

A randomized gradient-free attack on ReLU networks

Estimating Neural Network Robustness via Lipschitz Constant and Architecture Sensitivity

Lipschitz constant estimation for general neural network architectures using control tools

MIQCQP reformulation of the ReLU neural networks Lipschitz constant estimation problem

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

On the Complexity of Learning Neural Networks

Approximation Bounds for Norm Constrained Neural Networks with Applications to Regression and GANs