Abstract:The loss landscape of deep neural networks (DNNs) is commonly considered complex and wildly fluctuated. However, an interesting observation is that the loss surfaces plotted along Gaussian noise directions are almost v-basin ones with the perturbed model lying on the basin. This motivates us to rethink whether the 1D or 2D subspace could cover more complex local geometry structures, and how to mine the corresponding perturbation directions. This paper systematically and gradually categorizes the 1D curves from simple to complex, including v-basin, v-side, w-basin, w-peak, and vvv-basin curves. Notably, the latter two types are already hard to obtain via the intuitive construction of specific perturbation directions, and we need to propose proper mining algorithms to plot the corresponding 1D curves. Combining these 1D directions, various types of 2D surfaces are visualized such as the saddle surfaces and the bottom of a bottle of wine that are only shown by demo functions in previous works. Finally, we propose theoretical insights from the lens of the Hessian matrix to explain the observed several interesting phenomena.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is related to the complexity and visualization of the loss landscapes of deep neural networks (DNNs). Specifically, the paper focuses on the following aspects: 1. **Complexity of the loss landscape**: The traditional view is that the loss landscapes of DNNs are very complex and fluctuate violently. However, by observing the loss surfaces drawn along the Gaussian noise direction, it is found that these surfaces are almost in the shape of "v - basins", that is, the perturbed models are located within the basins. This finding prompts researchers to rethink whether one - dimensional or two - dimensional sub - spaces can capture more complex local geometric structures and how to mine the corresponding perturbation directions. 2. **Classification and mining of loss surfaces**: The paper systematically classifies one - dimensional curves from simple to complex, including types such as "v - basin", "v - side", "w - basin", "w - peak" and "vvv - basin". In particular, for curve types such as "w - basin" and "w - peak" that are difficult to obtain by intuitively constructing specific perturbation directions, appropriate mining algorithms are proposed to draw the corresponding one - dimensional curves. 3. **Low - dimensional visualization of high - dimensional loss landscapes**: Combining the above one - dimensional directions, the paper shows various types of two - dimensional surfaces, such as saddle - shaped surfaces and structures similar to the bottom of a wine bottle. These surfaces were usually only shown by simple example functions in previous literature, while this paper visualizes them through actual data. 4. **Theoretical explanation**: The paper provides theoretical insights from the perspective of the Hessian matrix to explain several interesting phenomena observed. In particular, the paper defines the phenomenon of monotonically increasing loss when perturbed by Gaussian noise (GMI) and provides a theoretical explanation from the perspective of the Hessian matrix. In addition, the paper also establishes a theoretical connection between GMI and the linear mode connection (LMC) phenomenon, providing a new perspective for explaining LMC. In summary, the main goal of this paper is to reveal the complex structures of DNNs' loss landscapes through systematic visualization and mining methods and to provide theoretical support to explain these phenomena. This not only helps to understand the training process of DNNs but also provides new ideas for the design of optimization algorithms.

Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks

Visualizing the Loss Landscape of Neural Nets

Better Loss Landscape Visualization for Deep Neural Networks with Trajectory Information.

Exploring the Geometry and Topology of Neural Network Loss Landscapes

Visualizing the Loss Landscape of Winning Lottery Tickets

Exploring Loss Landscapes through the Lens of Spin Glass Theory

Evaluating Loss Landscapes from a Topology Perspective

Visualizing high-dimensional loss landscapes with Hessian directions

Emergent properties of the local geometry of neural loss landscapes

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes.

Embedding Principle of Loss Landscape of Deep Neural Networks

Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

The Loss Surface of Deep Linear Networks Viewed Through the Algebraic Geometry Lens

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

Understanding Loss Landscapes of Neural Network Models in Solving Partial Differential Equations

Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes

Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks

On the curvature of the loss landscape

On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages