Abstract:Nonsmooth composite optimization with orthogonality constraints has a broad spectrum of applications in statistical learning and data science. However, this problem is generally challenging to solve due to its non-convex and non-smooth nature. Existing solutions are limited by one or more of the following restrictions: (i) they are full gradient methods that require high computational costs in each iteration; (ii) they are not capable of solving general nonsmooth composite problems; (iii) they are infeasible methods and can only achieve the feasibility of the solution at the limit point; (iv) they lack rigorous convergence guarantees; (v) they only obtain weak optimality of critical points. In this paper, we propose \textit{\textbf{OBCD}}, a new Block Coordinate Descent method for solving general nonsmooth composite problems under Orthogonality constraints. \textit{\textbf{OBCD}} is a feasible method with low computation complexity footprints. In each iteration, our algorithm updates $k$ rows of the solution matrix ($k\geq2$ is a parameter) to preserve the constraints. Then, it solves a small-sized nonsmooth composite optimization problem under orthogonality constraints either exactly or approximately. We demonstrate that any exact block-$k$ stationary point is always an approximate block-$k$ stationary point, which is equivalent to the critical stationary point. We are particularly interested in the case where $k=2$ as the resulting subproblem reduces to a one-dimensional nonconvex problem. We propose a breakpoint searching method and a fifth-order iterative method to solve this problem efficiently and effectively. We also propose two novel greedy strategies to find a good working set to further accelerate the convergence of \textit{\textbf{OBCD}}. Finally, we have conducted extensive experiments on several tasks to demonstrate the superiority of our approach.

A Splitting Method for Orthogonality Constrained Problems

Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints

Strong convergence of Bregman projection algorithms for solving split feasibility problems

A Feasible Method for Optimization with Orthogonality Constraints

Global Optimization with Orthogonality Constraints Via Stochastic Diffusion on Manifold

A splitting primal-dual proximity algorithm for solving composite optimization problems

Parallel inertial forward–backward splitting methods for solving variational inequality problems with variational inclusion constraints

Nonconvex and Nonsmooth Optimization with Generalized Orthogonality Constraints: an Approximate Augmented Lagrangian Method

A Forward-Backward Bregman Splitting Scheme for Regularized Distributed Optimization Problems

The splitting algorithms by Ryu, by Malitsky-Tam, and by Campoy applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints

Structured Quasi-Newton Methods for Optimization with Orthogonality Constraints.

A block Lanczos method for large-scale quadratic minimization problems with orthogonality constraints

A Peaceman–Rachford Splitting Method with Monotone Plus Skew-Symmetric Splitting for Nonlinear Saddle Point Problems

The splitting algorithms by Ryu and by Malitsky-Tam applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

Preconditioned Tensor Splitting AOR Iterative Methods for ℋ‐tensor Equations

An Efficient Matrix Splitting Method for the Second-Order Cone Complementarity Problem

Block Coordinate Descent Methods for Optimization under J-Orthogonality Constraints with Applications

Relaxed Inertial Methods for Solving Split Variational Inequality Problems Without Product Space Formulation

Strongly convergent inertial projection and contraction methods for split variational inequality problem

An inertial accelerated outer quadratic approximation method for split feasibility problem with application to elastic net