Abstract:We study optimization algorithms for the finite sum problems frequently arising in machine learning applications. First, we propose novel variants of stochastic gradient descent with a variance reduction property that enables linear convergence for strongly convex objectives. Second, we study distributed setting, in which the data describing the optimization problem does not fit into a single computing node. In this case, traditional methods are inefficient, as the communication costs inherent in distributed optimization become the bottleneck. We propose a communication-efficient framework which iteratively forms local subproblems that can be solved with arbitrary local optimization algorithms. Finally, we introduce the concept of Federated Optimization/Learning, where we try to solve the machine learning problems without having data stored in any centralized manner. The main motivation comes from industry when handling user-generated data. The current prevalent practice is that companies collect vast amounts of user data and store them in datacenters. An alternative we propose is not to collect the data in first place, and instead occasionally use the computational power of users' devices to solve the very same optimization problems, while alleviating privacy concerns at the same time. In such setting, minimization of communication rounds is the primary goal, and we demonstrate that solving the optimization problems in such circumstances is conceptually tractable.

Distributed Stochastic Optimization: Variance Reduction and Edge Based Method

Distributed Stochastic Algorithm for Global Optimization in Networked System

Distributed Stochastic Optimization under a General Variance Condition

Distributed Robust Optimization in Networked System.

Distributed Stochastic Optimization With Unbounded Subgradients Over Randomly Time-Varying Networks

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Distributed Optimization With Personalization: A Flexible Algorithmic Framework

Distributed Heterogeneous Multi-Agent Optimization with Stochastic Sub-Gradient

Distributed Optimization for Computation Offloading in Edge Computing

Stochastic, Distributed and Federated Optimization for Machine Learning

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Non-Smooth Setting of Stochastic Decentralized Convex Optimization Problem Over Time-Varying Graphs

Distributed Stochastic Strongly Convex Optimization under Heavy-Tailed Noises

Distributed Stochastic Subgradient Projection Algorithms for Convex Optimization

Distributed Optimization under Edge Agreement with Application in Battery Network Management

Convergence of Distributed Stochastic Variance Reduced Methods Without Sampling Extra Data

Discretized Distributed Optimization over Dynamic Digraphs

Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis

Numerical methods for distributed stochastic compositional optimization problems with aggregative structure

Distributed stochastic projection-free solver for constrained optimization

Distributed Stochastic Optimization with Random Communication and Computational Delays: Optimal Policies and Performance Analysis