Abstract:We develop a new randomized iterative algorithm---stochastic dual ascent (SDA)---for finding the projection of a given vector onto the solution space of a linear system. The method is dual in nature: with the dual being a non-strongly concave quadratic maximization problem without constraints. In each iteration of SDA, a dual variable is updated by a carefully chosen point in a subspace spanned by the columns of a random matrix drawn independently from a fixed distribution. The distribution plays the role of a parameter of the method. Our complexity results hold for a wide family of distributions of random matrices, which opens the possibility to fine-tune the stochasticity of the method to particular applications. We prove that primal iterates associated with the dual process converge to the projection exponentially fast in expectation, and give a formula and an insightful lower bound for the convergence rate. We also prove that the same rate applies to dual function values, primal function values and the duality gap. Unlike traditional iterative methods, SDA converges under no additional assumptions on the system (e.g., rank, diagonal dominance) beyond consistency. In fact, our lower bound improves as the rank of the system matrix drops. Many existing randomized methods for linear systems arise as special cases of SDA, including randomized Kaczmarz, randomized Newton, randomized coordinate descent, Gaussian descent, and their variants. In special cases where our method specializes to a known algorithm, we either recover the best known rates, or improve upon them. Finally, we show that the framework can be applied to the distributed average consensus problem to obtain an array of new algorithms. The randomized gossip algorithm arises as a special case.

A new dual spectral projected gradient method for log-determinant semidefinite programming with hidden clustering structures

Dual Spectral Projected Gradient Method for Generalized Log-det Semidefinite Programming

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Efficient Semidefinite Spectral Clustering Via Lagrange Duality.

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

A New Nonmonotone Spectral Projected Gradient Algorithm for Box-Constrained Optimization Problems in M×n Real Matrix Space with Application in Image Clustering.

Proximal Gradient/Semismooth Newton Methods for Projection Onto a Polyhedron Via the Duality-Gap-Active-Set Strategy

Half-Space Proximal Stochastic Gradient Method for Group-Sparsity Regularized Problem

A Semismooth Newton-CG Based Dual PPA for Matrix Spectral Norm Approximation Problems

Solving Log-Determinant Optimization Problems by a Newton-CG Primal Proximal Point Algorithm

A Low-rank Projected Proximal Gradient Method for Spectral Compressed Sensing

Fully Stochastic Primal-dual Gradient Algorithm for Non-convex Optimization on Random Graphs

Projected Gradient Descent for Spectral Compressed Sensing via Symmetric Hankel Factorization

A conditional gradient homotopy method with applications to Semidefinite Programming

Nonconvex Sparse Spectral Clustering by Alternating Direction Method of Multipliers and Its Convergence Analysis

Efficient Projected Gradient Methods for Cardinality Constrained Optimization

Stochastic Dual Ascent for Solving Linear Systems

Subspace Clustering Using Log-determinant Rank Approximation

Semidefinite programming on population clustering: a global analysis

Fast Admm For Homogeneous Self-Dual Embedding Of Sparse Sdps

Debiasing and a local analysis for population clustering using semidefinite programming