Abstract:The optimization problem associated to fitting two-layer ReLU networks having $d$~inputs, $k$~neurons, and labels generated by a target network, is considered. Two types of infinite families of spurious minima, giving one minimum per $d$, were recently found. The loss at minima belonging to the first type converges to zero as $d$ increases. In the second type, the loss remains bounded away from zero. That being so, how may one avoid minima belonging to the latter type? Fortunately, such minima are never detected by standard optimization methods. Motivated by questions concerning the nature of this phenomenon, we develop methods to study distinctive analytic properties of hidden minima. By existing analyses, the Hessian spectrum of both types agree modulo $O(d^{-1/2})$-terms -- not promising. Thus, rather, our investigation proceeds by studying curves along which the loss is minimized or maximized, generally referred to as tangency arcs. We prove that apparently far removed group representation-theoretic considerations concerning the arrangement of subspaces invariant to the action of subgroups of $S_d$, the symmetry group over $d$ symbols, relative to ones fixed by the action yield a precise description of all finitely many admissible types of tangency arcs. The general results used for the loss function reveal that arcs emanating from hidden minima differ, characteristically, by their structure and symmetry, precisely on account of the $O(d^{-1/2})$-eigenvalue terms absent in previous work, indicating in particular the subtlety of the analysis. The theoretical results, stated and proved for o-minimal structures, show that the set comprising all tangency arcs is topologically sufficiently tame to enable a numerical construction of tangency arcs and so compare how minima, both types, are positioned relative to adjacent critical points.

Spurious Local Minima Provably Exist for Deep Convolutional Neural Networks

Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations

On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems

Avoiding Spurious Local Minima in Deep Quadratic Networks

Spurious Local Minima of Deep ReLU Neural Networks in the Neural Tangent Kernel Regime

Exact Solutions of a Deep Linear Network

Elimination of All Bad Local Minima in Deep Learning

A Study on the Uncertainty of Convolutional Layers in Deep Neural Networks

Depth Creates No Bad Local Minima

Piecewise linear activations substantially shape the loss surfaces of neural networks

On the existence of minimizers in shallow residual ReLU neural network optimization landscapes

Neural Networks with Complex-Valued Weights Have No Spurious Local Minima

Hidden Minima in Two-Layer ReLU Networks

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

Asymmetric Valleys: Beyond Sharp and Flat Local Minima.

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes

Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

Are deep ResNets provably better than linear predictors?

Dynamics in Deep Classifiers Trained with the Square Loss: Normalization, Low Rank, Neural Collapse, and Generalization Bounds

Effect of Depth and Width on Local Minima in Deep Learning