Abstract:In this paper, we explicitly determine local and global minimizers of the $\mathcal{L}^2$ cost function in underparametrized Deep Learning (DL) networks; our main goal is to shed light on their geometric structure and properties. We accomplish this by a direct construction, without invoking the gradient descent flow at any point of this work. We specifically consider $L$ hidden layers, a ReLU ramp activation function, an $\mathcal{L}^2$ Schatten class (or Hilbert-Schmidt) cost function, input and output spaces $\mathbb{R}^Q$ with equal dimension $Q\geq1$, and hidden layers also defined on $\mathbb{R}^{Q}$; the training inputs are assumed to be sufficiently clustered. The training input size $N$ can be arbitrarily large - thus, we are considering the underparametrized regime. More general settings are left to future work. We construct an explicit family of minimizers for the global minimum of the cost function in the case $L\geq Q$, which we show to be degenerate. Moreover, we determine a set of $2^Q-1$ distinct degenerate local minima of the cost function. In the context presented here, the concatenation of hidden layers of the DL network is reinterpreted as a recursive application of a {\em truncation map} which "curates" the training inputs by minimizing their noise to signal ratio.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to understand the geometric structure and properties of local and global minima of the loss function in under - parameterized deep learning (DL) networks through strict mathematical methods. Specifically, the author hopes to explicitly determine these minima without relying on the gradient descent flow and explain their degeneracy characteristics. ### Core problems of the paper 1. **Explicitly determine local and global minima**: - The author's goal is to explicitly determine the local and global minima in under - parameterized DL networks by means of direct construction rather than using the gradient - descent method. 2. **Understand the geometric structure and properties of minima**: - The author hopes to understand the geometric structure and properties of these minima through strict mathematical derivations, especially how to construct global minima when the number of hidden layers $ L \geq Q $. 3. **Handle training inputs under specific conditions**: - The research assumes that the training data is sufficiently clustered, that is, the clusters of training inputs can be effectively "pruned" to reduce the ratio of noise to signal. ### Specific settings and assumptions - **Network structure**: Consider a DL network with $ L $ hidden layers, and each hidden layer uses the ReLU activation function. - **Loss function**: Adopt the $ L_2 $ Schatten - class (or Hilbert - Schmidt) loss function. - **Input - output space**: The dimensions of both the input space and the output space are $ Q $, and the hidden layers are also defined on $ R^Q $. - **Training input size**: The number of training inputs $ N $ can be arbitrarily large, so it belongs to the under - parameterized regime. - **Training data assumption**: Assume that the training data is sufficiently clustered, which helps to simplify the problem and clearly show the key principles. ### Main contributions - **Construction of global minima**: For the case of $ L \geq Q $, the author explicitly constructs a family of global minima and proves that they are degenerate. - **Determination of local minima**: The author also determines $ 2Q - 1 $ different degenerate local minima. - **New interpretation of hidden layers**: Re - interpret the hidden layers as the application of a series of truncation mappings, which can "prune" the training inputs, thereby reducing the ratio of noise to signal. ### Conclusion Through these studies, the author provides a profound mathematical understanding of the minima of the loss function in DL networks, especially in the under - parameterized case, revealing the geometric structure and degeneracy characteristics of these minima. This is of great significance for further understanding the working mechanism of DL networks and their optimization processes.

Geometric structure of Deep Learning networks and construction of global ${\mathcal L}^2$ minimizers

Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization

Global $\mathcal{L}^2$ minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning

Interpretable global minima of deep ReLU neural networks on sequentially separable data

Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

Hidden Minima in Two-Layer ReLU Networks

Exact Solutions of a Deep Linear Network

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

Training a Two Layer ReLU Network Analytically

Avoiding Spurious Local Minima in Deep Quadratic Networks

Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes

The Loss Surface of Deep Linear Networks Viewed Through the Algebraic Geometry Lens

Theoretical properties of the global optimizer of two layer neural network

Cost-Sensitive Deep Learning with Layer-Wise Cost Estimation

Dynamics in Deep Classifiers Trained with the Square Loss: Normalization, Low Rank, Neural Collapse, and Generalization Bounds

On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems

The geometry of the deep linear network

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

The loss landscape of deep linear neural networks: a second-order analysis

The Geometric Structure of Fully-Connected ReLU Layers