Abstract:Nonsmooth composite optimization with orthogonality constraints has a broad spectrum of applications in statistical learning and data science. However, this problem is generally challenging to solve due to its non-convex and non-smooth nature. Existing solutions are limited by one or more of the following restrictions: (i) they are full gradient methods that require high computational costs in each iteration; (ii) they are not capable of solving general nonsmooth composite problems; (iii) they are infeasible methods and can only achieve the feasibility of the solution at the limit point; (iv) they lack rigorous convergence guarantees; (v) they only obtain weak optimality of critical points. In this paper, we propose \textit{\textbf{OBCD}}, a new Block Coordinate Descent method for solving general nonsmooth composite problems under Orthogonality constraints. \textit{\textbf{OBCD}} is a feasible method with low computation complexity footprints. In each iteration, our algorithm updates $k$ rows of the solution matrix ($k\geq2$ is a parameter) to preserve the constraints. Then, it solves a small-sized nonsmooth composite optimization problem under orthogonality constraints either exactly or approximately. We demonstrate that any exact block-$k$ stationary point is always an approximate block-$k$ stationary point, which is equivalent to the critical stationary point. We are particularly interested in the case where $k=2$ as the resulting subproblem reduces to a one-dimensional nonconvex problem. We propose a breakpoint searching method and a fifth-order iterative method to solve this problem efficiently and effectively. We also propose two novel greedy strategies to find a good working set to further accelerate the convergence of \textit{\textbf{OBCD}}. Finally, we have conducted extensive experiments on several tasks to demonstrate the superiority of our approach.

A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints

Block Coordinate Descent Methods for Optimization under J-Orthogonality Constraints with Applications

Block Coordinate Descent Methods for Structured Nonconvex Optimization with Nonseparable Constraints: Optimality Conditions and Global Convergence

On the Flexibility of Block Coordinate Descent for Large-Scale Optimization.

Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

A Block Decomposition Algorithm for Sparse Optimization

Efficiency of Coordinate Descent Methods For Structured Nonconvex Optimization

A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

A Flexible Coordinate Descent Method

Stochastic Primal-Dual Coordinate Method for Nonlinear Convex Cone Programs

Robust Block Coordinate Descent

Synchronous Parallel Block Coordinate Descent Method for Nonsmooth Convex Function Minimization

Randomized block coordinate descent method for linear ill-posed problems

Parallel Distributed Block Coordinate Descent Methods based on Pairwise Comparison Oracle

Hybrid Jacobian and Gauss-Seidel proximal block coordinate update methods for linearly constrained convex programming

Acceleration of block coordinate descent method achieves the O(1/k2) rate of convergence for a convex function with block coordinate strong convexity

A Parallel Line Search Subspace Correction Method for Composite Convex Optimization

Block Coordinate Descentwith Time Perturbation for Nonconvex Nonsmooth Problems in Real-World Studies

Randomized Coordinate Subgradient Method for Nonsmooth Composite Optimization

Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints