Abstract:Nonconvex optimization arises in many areas of computational science and engineering. However, most nonconvex optimization algorithms are only known to have local convergence or subsequence convergence properties. In this paper, we propose an algorithm for nonconvex optimization and establish its global convergence (of the whole sequence) to a critical point. In addition, we give its asymptotic convergence rate and numerically demonstrate its efficiency. In our algorithm, the variables of the underlying problem are either treated as one block or multiple disjoint blocks. It is assumed that each non-differentiable component of the objective function, or each constraint, applies only to one block of variables. The differentiable components of the objective function, however, can involve multiple blocks of variables together. Our algorithm updates one block of variables at a time by minimizing a certain prox-linear surrogate, along with an extrapolation to accelerate its convergence. The order of update can be either deterministically cyclic or randomly shuffled for each cycle. In fact, our convergence analysis only needs that each block be updated at least once in every fixed number of iterations. We show its global convergence (of the whole sequence) to a critical point under fairly loose conditions including, in particular, the Kurdyka–Łojasiewicz condition, which is satisfied by a broad class of nonconvex/nonsmooth applications. These results, of course, remain valid when the underlying problem is convex. We apply our convergence results to the coordinate descent iteration for non-convex regularized linear regression, as well as a modified rank-one residue iteration for nonnegative matrix factorization. We show that both applications have global convergence. Numerically, we tested our algorithm on nonnegative matrix and tensor factorization problems, where random shuffling clearly improves the chance to avoid low-quality local solutions.

Block Coordinate Descent Methods for Structured Nonconvex Optimization with Nonseparable Constraints: Optimality Conditions and Global Convergence

On the Flexibility of Block Coordinate Descent for Large-Scale Optimization.

Efficiency of Coordinate Descent Methods For Structured Nonconvex Optimization

A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints

Markov Chain Block Coordinate Descent.

Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization

Block Coordinate Descent Methods for Optimization under J-Orthogonality Constraints with Applications

Stochastic Primal-Dual Coordinate Method for Nonlinear Convex Cone Programs

Randomized block coordinate descent method for linear ill-posed problems

Robust Block Coordinate Descent

Extended ADMM and BCD for Nonseparable Convex Minimization Models with Quadratic Coupling Terms: Convergence Analysis and Insights

A Flexible Coordinate Descent Method

Block Stochastic Gradient Iteration for Convex and Nonconvex Optimization.

Stochastic Coordinate Descent Methods for Regularized Smooth and Nonsmooth Losses.

Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks

A Globally Convergent Algorithm for Nonconvex Optimization Based on Block Coordinate Update

A Block Coordinate Descent Method For Regularized Multiconvex Optimization With Applications To Nonnegative Tensor Factorization And Completion

Stochastic Parallel Block Coordinate Descent for Large-scale Saddle Point Problems

Block Coordinate Descent Only Converge to Minimizers

Coordinate Descent for MCP/SCAD Penalized Least Squares Converges Linearly

WITHDRAWN: Parallel Block Coordinate Descent with Path-Following for Constrained Folded Concave Linear Regression