Abstract:Local SGD, a cornerstone algorithm in federated learning, is widely used in training deep neural networks and shown to have strong empirical performance. A theoretical understanding of such performance on nonconvex loss landscapes is currently lacking. Analysis of the global convergence of SGD is challenging, as the noise depends on the model parameters. Indeed, many works narrow their focus to GD and rely on injecting noise to enable convergence to the local or global optimum. When expanding the focus to local SGD, existing analyses in the nonconvex case can only guarantee finding stationary points or assume the neural network is overparameterized so as to guarantee convergence to the global minimum through neural tangent kernel analysis. In this work, we provide the first global convergence analysis of the vanilla local SGD for two-layer neural networks \emph{without overparameterization} and \textit{without injecting noise}, when the input data is Gaussian. The main technical ingredients of our proof are \textit{a self-correction mechanism} and \textit{a new exact recursive characterization of the direction of global model parameters}. The self-correction mechanism guarantees the algorithm reaches a good region even if the initialization is in a bad region. A good (bad) region means updating the model by gradient descent will move closer to (away from) the optimal solution. The main difficulty in establishing a self-correction mechanism is to cope with the gradient dependency between two layers. To address this challenge, we divide the landscape of the objective into several regions to carefully control the interference of two layers during the correction process. As a result, we show that local SGD can correct the two layers and enter the good region in polynomial time. After that, we establish a new exact recursive characterization of the direction of global parameters, which is the key to showing convergence to the global minimum with linear speedup in the number of machines and reduced communication rounds. Experiments on synthetic data confirm theoretical results.

Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks With Soft-Thresholding

Dual-Focal Camera Hdr Imaging Based On Convolutional Neural Network

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Exploring the loss landscape of regularized neural networks via convex duality

On the Learning Dynamics of Two-layer Nonlinear Convolutional Neural Networks.

Convex Formulation of Overparameterized Deep Neural Networks

Dual Convolutional Neural Networks for Low-Level Vision

Convex Formulations for Training Two-Layer ReLU Neural Networks

A Layer-Wise Theoretical Framework for Deep Learning of Convolutional Neural Networks

A Two-Layer Recurrent Neural Network for Nonsmooth Convex Optimization Problems

The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models

Dual-channel deep graph convolutional neural networks

Parameter Convex Neural Networks

DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks

Doubly Convolutional Neural Networks

On the learning dynamics of two-layer quadratic neural networks for understanding deep learning

Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time

Convolutional Networks with Dense Connectivity

Global Convergence Analysis of Local SGD for Two-layer Neural Network Without Overparameterization

The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program

Lossless Convexification and Duality