Abstract: Finding a solution to the linear system $Ax = b$ with various minimization properties arises from many engineering and computer science applications, including compressed sensing, image processing, and machine learning. In the age of big data, the scalability of stochastic optimization algorithms has made it increasingly important to solve problems of unprecedented sizes. This paper focuses on the problem of minimizing a strongly convex objective function subject to linearly constraints. We consider the dual formulation of this problem and adopt the stochastic coordinate descent to solve it. The proposed algorithmic framework, called fast stochastic dual coordinate descent, utilizes an adaptive variation of Polyak's heavy ball momentum and user-defined distributions for sampling. Our adaptive heavy ball momentum technique can efficiently update the parameters by using iterative information, overcoming the limitation of the heavy ball momentum method where prior knowledge of certain parameters, such as singular values of a matrix, is required. We prove that, under strongly admissible of the objective function, the propose method converges linearly in expectation. By varying the sampling matrix, we recover a comprehensive array of well-known algorithms as special cases, including the randomized sparse Kaczmarz method, the randomized regularized Kaczmarz method, the linearized Bregman iteration, and a variant of the conjugate gradient (CG) method. Numerical experiments are provided to confirm our results.

A Fast Coordinate Descent Method for High-Dimensional Non-Negative Least Squares using a Unified Sparse Regression Framework

A Fast Coordinate Descent Method for High-Dimensional Non-Negative Least Squares using a Unified Sparse Regression Framework

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

A Fast and Scalable Pathwise-Solver for Group Lasso and Elastic Net Penalized Regression via Block-Coordinate Descent

Fast Sparse Least-Squares Regression with Non-Asymptotic Guarantees

A Constructive Approach to High-dimensional Regression

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Coordinate Descent for MCP/SCAD Penalized Least Squares Converges Linearly

Generalized Newton–Raphson Algorithm for High Dimensional LASSO Regression

Fast stochastic dual coordinate descent algorithms for linearly constrained convex optimization

A Coordinate-wise Optimization Algorithm for Sparse Inverse Covariance Selection.

Row-Sparse Principal Component Analysis via Coordinate Descent Method

Generalized Low-rank plus Sparse Tensor Estimation by Fast Riemannian Optimization

RESEARCH ARTICLE A Coordinate Majorization Descent Algorithm for ℓ1 Penalized Learning

Accelerated Sparse Linear Regression Via Random Projection.

Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling

High-Dimensional Sparse Data Low-rank Representation via Accelerated Asynchronous Parallel Stochastic Gradient Descent

Semismooth Newton Coordinate Descent Algorithm for Elastic-Net Penalized Huber Loss Regression and Quantile Regression

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

Fast and Adaptive Sparse Precision Matrix Estimation in High Dimensions

A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data