Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Speeding up COMPASS for high-dimensional discrete optimization via simulation

An Adaptive Hyperbox Algorithm for High-Dimensional Discrete Optimization Via Simulation Problems

Discrete Optimization Via Simulation Using Coordinate Search

Discrete optimization via simulation: algorithms and error control

Adaptive Partitioning Strategy for High-Dimensional Discrete Simulation-based Optimization Problems

A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling

An adaptive sampling algorithm for simulation-based optimization with descriptive complexity constraints

Fast stochastic dual coordinate descent algorithms for linearly constrained convex optimization

High-Dimensional Simulation Optimization Via Brownian Fields and Sparse Grids

Industrial Strength COMPASS

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Simulation-Based Discrete Optimization of Stochastic Discrete Event Systems Subject to Non Closed-Form Constraints.

Determining the Optimal Sampling Set Size for Random Search

A Framework For Locally Convergent Random-Search Algorithms For Discrete Optimization Via Simulation

When is the Convergence Time of Langevin Algorithms Dimension Independent? A Composite Optimization Viewpoint

A novel multi-objective compass search

The Search direction Correction makes first-order methods faster

Convergence Acceleration of Markov Chain Monte Carlo-based Gradient Descent by Deep Unfolding

A Consensus-Based Global Optimization Method for High Dimensional Machine Learning Problems

A novel batch-selection strategy for parallel global optimization