Abstract:Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradient method that randomly updates block-coordinate variables by using block composite subgradient mapping. This method converges asymptotically to critical points with proven sublinear convergence rate for certain optimality measures. Second, we develop a randomly permuted CD method with two alternating steps: linearizing the concave part and cycling through variables. We prove asymptotic convergence to critical points and sublinear complexity rate for objectives with both smooth and concave parts. Third, we extend accelerated coordinate descent (ACD) to nonsmooth and nonconvex optimization to develop a novel randomized proximal DC algorithm whereby we solve the subproblem inexactly by ACD. Convergence is guaranteed with at most a few number of ACD iterations for each DC subproblem, and convergence complexity is established for identification of some approximate critical points. Fourth, we further develop the third method to minimize certain ill-conditioned nonconvex functions: weakly convex functions with high Lipschitz constant to negative curvature ratios. We show that, under specific criteria, the ACD-based randomized method has superior complexity compared to conventional gradient methods. Finally, an empirical study on sparsity-inducing learning models demonstrates that CD methods are superior to gradient-based methods for certain large-scale problems.

Coordinate Descent with Bandit Sampling

Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedback

Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexity

Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling

A Flexible Coordinate Descent Method

A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization

Decomposition-Coordination Method for Finite Horizon Bandit Problems

Randomized Dual Coordinate Ascent with Arbitrary Sampling

Carathéodory Sampling for Stochastic Gradient Descent

Inexact Coordinate Descent: Complexity and Preconditioning

Coordinate-Update Algorithms can Efficiently Detect Infeasible Optimization Problems

Regret Minimization and Statistical Inference in Online Decision Making with High-dimensional Covariates

Efficient and Adaptive Posterior Sampling Algorithms for Bandits

On convergence of a $q$-random coordinate constrained algorithm for non-convex problems

Dykstra's Algorithm, ADMM, and Coordinate Descent: Connections, Insights, and Extensions

Hamiltonian Descent and Coordinate Hamiltonian Descent

Robust Block Coordinate Descent

Greedy coordinate descent from the view of $\ell_1$-norm gradient descent

Randomness and Permutations in Coordinate Descent Methods

Efficiency of Coordinate Descent Methods For Structured Nonconvex Optimization