Abstract:Local SGD, a cornerstone algorithm in federated learning, is widely used in training deep neural networks and shown to have strong empirical performance. A theoretical understanding of such performance on nonconvex loss landscapes is currently lacking. Analysis of the global convergence of SGD is challenging, as the noise depends on the model parameters. Indeed, many works narrow their focus to GD and rely on injecting noise to enable convergence to the local or global optimum. When expanding the focus to local SGD, existing analyses in the nonconvex case can only guarantee finding stationary points or assume the neural network is overparameterized so as to guarantee convergence to the global minimum through neural tangent kernel analysis. In this work, we provide the first global convergence analysis of the vanilla local SGD for two-layer neural networks \emph{without overparameterization} and \textit{without injecting noise}, when the input data is Gaussian. The main technical ingredients of our proof are \textit{a self-correction mechanism} and \textit{a new exact recursive characterization of the direction of global model parameters}. The self-correction mechanism guarantees the algorithm reaches a good region even if the initialization is in a bad region. A good (bad) region means updating the model by gradient descent will move closer to (away from) the optimal solution. The main difficulty in establishing a self-correction mechanism is to cope with the gradient dependency between two layers. To address this challenge, we divide the landscape of the objective into several regions to carefully control the interference of two layers during the correction process. As a result, we show that local SGD can correct the two layers and enter the good region in polynomial time. After that, we establish a new exact recursive characterization of the direction of global parameters, which is the key to showing convergence to the global minimum with linear speedup in the number of machines and reduced communication rounds. Experiments on synthetic data confirm theoretical results.

Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network

Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization

Exploring the Geometry and Topology of Neural Network Loss Landscapes

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes.

The Loss Surface of Deep Linear Networks Viewed Through the Algebraic Geometry Lens

On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems

Global Convergence Analysis of Local SGD for Two-layer Neural Network Without Overparameterization

Emergent properties of the local geometry of neural loss landscapes

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes

Loss Landscape Characterization of Neural Networks without Over-Parametrization

Visualizing the Loss Landscape of Neural Nets

A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

Taxonomizing local versus global structure in neural network loss landscapes

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit

Global $\mathcal{L}^2$ minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning

Asymmetric Valleys: Beyond Sharp and Flat Local Minima.