Abstract:We develop a new randomized iterative algorithm---stochastic dual ascent (SDA)---for finding the projection of a given vector onto the solution space of a linear system. The method is dual in nature: with the dual being a non-strongly concave quadratic maximization problem without constraints. In each iteration of SDA, a dual variable is updated by a carefully chosen point in a subspace spanned by the columns of a random matrix drawn independently from a fixed distribution. The distribution plays the role of a parameter of the method. Our complexity results hold for a wide family of distributions of random matrices, which opens the possibility to fine-tune the stochasticity of the method to particular applications. We prove that primal iterates associated with the dual process converge to the projection exponentially fast in expectation, and give a formula and an insightful lower bound for the convergence rate. We also prove that the same rate applies to dual function values, primal function values and the duality gap. Unlike traditional iterative methods, SDA converges under no additional assumptions on the system (e.g., rank, diagonal dominance) beyond consistency. In fact, our lower bound improves as the rank of the system matrix drops. Many existing randomized methods for linear systems arise as special cases of SDA, including randomized Kaczmarz, randomized Newton, randomized coordinate descent, Gaussian descent, and their variants. In special cases where our method specializes to a known algorithm, we either recover the best known rates, or improve upon them. Finally, we show that the framework can be applied to the distributed average consensus problem to obtain an array of new algorithms. The randomized gossip algorithm arises as a special case.

Stochastic gradient descent for linear least squares problems with partially observed data

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Linearly convergent adjoint free solution of least squares problems by random descent

Unbiased least squares regression via averaged stochastic gradient descent

Greedy Double Subspaces Coordinate Descent Method for Solving Linear Least-Squares Problems

Stochastic Sub-Sampled Newton Method with Variance Reduction

Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation

On adaptive stochastic extended iterative methods for solving least squares

A doubly stochastic block Gauss-Seidel algorithm for solving linear equations

Stochastic Dual Ascent for Solving Linear Systems

Randomized subspace gradient method for constrained optimization

Constrained Stochastic Gradient Descent for Large-Scale Least Squares Problem

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

Supervised Descent Method for Solving Nonlinear Least Squares Problems in Computer Vision

Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Stochastic Gradient Descent Method with Convex Penalty for Ill-Posed Problems in Banach Spaces

Application of Stochastic Gradient Descent Technique for Method of Moments

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

Randomized symmetric Gauss-Seidel method for solving linear least squares problems

Optimal Randomized First-Order Methods for Least-Squares Problems