Abstract:It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume that the activation function is ReLU or they crucially depend on the minimum eigenvalue of a certain Gram matrix depending on the data, random initialization and the activation function. In the later case, existing works only prove that this minimum eigenvalue is non-zero and do not provide quantitative bounds. On the empirical side, a contemporary line of investigations has proposed a number of alternative activation functions which tend to perform better than ReLU at least in some settings but no clear understanding has emerged. This state of affairs underscores the importance of theoretically understanding the impact of activation functions on training. In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks. A crucial property that governs the performance of an activation is whether or not it is smooth. For non-smooth activations such as ReLU, SELU and ELU, all eigenvalues of the associated Gram matrix are large under minimal assumptions on the data. For smooth activations such as tanh, swish and polynomials, the situation is more complex. If the subspace spanned by the data has small dimension then the minimum eigenvalue of the Gram matrix can be small leading to slow training. But if the dimension is large and the data satisfies another mild condition, then the eigenvalues are large. If we allow deep networks, then the small data dimension is not a limitation provided that the depth is sufficient. We discuss a number of extensions and applications of these results.

Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant

Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures

1-Lipschitz Neural Networks are more expressive with N-Activations

A Variational Inequality Model for Learning Neural Networks

Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths

Learning continuous piecewise non-linear activation functions for deep neural networks

Activation function optimization method: Learnable series linear units (LSLUs)

Effect of Activation Functions on the Training of Overparametrized Neural Nets

Trainable Activation Function in Image Classification

Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL

Variational Neural Networks: Every Layer and Neuron Can Be Unique

Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning

Neural networks with trainable matrix activation functions

Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants

On the Complexity of Learning Neural Networks

Delayed Recurrent Neural Networks with Global Lipschitz Activation Functions

Lipschitz constant estimation for general neural network architectures using control tools

A Unified Algebraic Perspective on Lipschitz Neural Networks

ReLUs Are Sufficient for Learning Implicit Neural Representations