No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths

Charles Guille-Escuret,Hiroki Naganuma,Kilian Fatras,Ioannis Mitliagkas

2023-06-21

Abstract:Understanding the optimization dynamics of neural networks is necessary for closing the gap between theory and practice. Stochastic first-order optimization algorithms are known to efficiently locate favorable minima in deep neural networks. This efficiency, however, contrasts with the non-convex and seemingly complex structure of neural loss landscapes. In this study, we delve into the fundamental geometric properties of sampled gradients along optimization paths. We focus on two key quantities, which appear in the restricted secant inequality and error bound. Both hold high significance for first-order optimization. Our analysis reveals that these quantities exhibit predictable, consistent behavior throughout training, despite the stochasticity induced by sampling minibatches. Our findings suggest that not only do optimization trajectories never encounter significant obstacles, but they also maintain stable dynamics during the majority of training. These observed properties are sufficiently expressive to theoretically guarantee linear convergence and prescribe learning rate schedules mirroring empirical practices. We conduct our experiments on image classification, semantic segmentation and language modeling across different batch sizes, network architectures, datasets, optimizers, and initialization seeds. We discuss the impact of each factor. Our work provides novel insights into the properties of neural network loss functions, and opens the door to theoretical frameworks more relevant to prevalent practice.

Machine Learning,Optimization and Control

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the contradiction between the practical efficiency and the theoretically complex loss landscape encountered during neural network optimization. Specifically:

1. **Understanding Optimization Dynamics**: The paper attempts to understand the dynamic characteristics of the neural network optimization process by analyzing the geometric properties of the optimization path. Despite the theoretically very complex loss function of neural networks (non-convex and potentially containing bad local minima and saddle points), actual training exhibits highly efficient optimization performance.

2. **Quantifying Simplified Characteristics**: Researchers have found that the actual loss landscape of neural networks is much simpler than theoretically expected. The paper attempts to quantify this simplified characteristic and proposes a new method to characterize these properties.

3. **Verifying Stable Patterns**: The paper focuses on two key quantities—Restricted Secant Inequality (RSI) and Error Bound (EB). These two quantities exhibit predictable and stable patterns during the stochastic gradient descent process, even when using small batch samples.

4. **Consistency Between Theory and Practice**: The paper attempts to integrate these observations with existing theoretical frameworks to better explain the optimization phenomena in practical applications and propose reasonable learning rate adjustment strategies.

Through these studies, the paper provides new insights into the characteristics of neural network loss functions and opens the door for further theoretical research.