Abstract:A variety of recent works, spanning pruning, lottery tickets, and training within random subspaces, have shown that deep neural networks can be trained using far fewer degrees of freedom than the total number of parameters. We analyze this phenomenon for random subspaces by first examining the success probability of hitting a training loss sub-level set when training within a random subspace of a given training dimensionality. We find a sharp phase transition in the success probability from $0$ to $1$ as the training dimension surpasses a threshold. This threshold training dimension increases as the desired final loss decreases, but decreases as the initial loss decreases. We then theoretically explain the origin of this phase transition, and its dependence on initialization and final desired loss, in terms of properties of the high-dimensional geometry of the loss landscape. In particular, we show via Gordon's escape theorem, that the training dimension plus the Gaussian width of the desired loss sub-level set, projected onto a unit sphere surrounding the initialization, must exceed the total number of parameters for the success probability to be large. In several architectures and datasets, we measure the threshold training dimension as a function of initialization and demonstrate that it is a small fraction of the total parameters, implying by our theory that successful training with so few dimensions is possible precisely because the Gaussian width of low loss sub-level sets is very large. Moreover, we compare this threshold training dimension to more sophisticated ways of reducing training degrees of freedom, including lottery tickets as well as a new, analogous method: lottery subspaces. Code is available at <a class="link-external link-https" href="https://github.com/ganguli-lab/degrees-of-freedom" rel="external noopener nofollow">this https URL</a>.

On Connected Sublevel Sets in Deep Learning

A Note on Connectivity of Sublevel Sets in Deep Learning

Avoiding Spurious Local Minima in Deep Quadratic Networks

Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems

On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems

Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding

The Loss Surface of Deep Linear Networks Viewed Through the Algebraic Geometry Lens

Piecewise linear activations substantially shape the loss surfaces of neural networks

Elimination of All Bad Local Minima in Deep Learning

Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations

Asymmetric Valleys: Beyond Sharp and Flat Local Minima.

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes.

Depth Creates No Bad Local Minima

Topological obstruction to the training of shallow ReLU neural networks

How many degrees of freedom do we need to train deep networks: a loss landscape perspective

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

A theoretical framework for deep locally connected ReLU network

Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

Embedding Principle of Loss Landscape of Deep Neural Networks