Abstract:Nonsmooth composite optimization with orthogonality constraints has a broad spectrum of applications in statistical learning and data science. However, this problem is generally challenging to solve due to its non-convex and non-smooth nature. Existing solutions are limited by one or more of the following restrictions: (i) they are full gradient methods that require high computational costs in each iteration; (ii) they are not capable of solving general nonsmooth composite problems; (iii) they are infeasible methods and can only achieve the feasibility of the solution at the limit point; (iv) they lack rigorous convergence guarantees; (v) they only obtain weak optimality of critical points. In this paper, we propose \textit{\textbf{OBCD}}, a new Block Coordinate Descent method for solving general nonsmooth composite problems under Orthogonality constraints. \textit{\textbf{OBCD}} is a feasible method with low computation complexity footprints. In each iteration, our algorithm updates $k$ rows of the solution matrix ($k\geq2$ is a parameter) to preserve the constraints. Then, it solves a small-sized nonsmooth composite optimization problem under orthogonality constraints either exactly or approximately. We demonstrate that any exact block-$k$ stationary point is always an approximate block-$k$ stationary point, which is equivalent to the critical stationary point. We are particularly interested in the case where $k=2$ as the resulting subproblem reduces to a one-dimensional nonconvex problem. We propose a breakpoint searching method and a fifth-order iterative method to solve this problem efficiently and effectively. We also propose two novel greedy strategies to find a good working set to further accelerate the convergence of \textit{\textbf{OBCD}}. Finally, we have conducted extensive experiments on several tasks to demonstrate the superiority of our approach.

A Block Incomplete Orthogonalization Method for Large Nonsymmetric Eigenproblems

On IOM(q): the Incomplete Orthogonalization Method for Large Unsymmetric Linear Systems

Some Recursions on Arnoldi's Method and IOM for Large Non-Hermitian Linear Systems

Generalized Block Lanczos Methods for Large Unsymmetric Eigenproblems

A Variation on the Block Arnoldi Method for Large Unsymmetric Matrix Eigenproblems

A generalized skew-symmetric Lanczos bidiagonalization method for computing several extreme eigenpairs of a large skew-symmetric/symmetric positive definite matrix pair

A Refined Iterative Algorithm Based on the Block Arnoldi Process for Large Unsymmetric Eigenproblems

The Second-Order Biorthogonalization Procedure and Its Application to Quadratic Eigenvalue Problems

A Convergence Analysis of the Inexact Rayleigh Quotient Iteration and Simplified Jacobi-Davidson Method for the Large Hermitian Matrix Eigenproblem

On IGMRES: an Incomplete Generalized Minimal Residual Method for Large Unsymmetric Linear Systems

A Refined Block Conjugate Gradient Method for Symmetric Eigenvalue Problems

A positivity preserving inexact Noda iteration for computing the smallest eigenpair of a large irreducible M -matrix

Some Problems in Large Scale Non-Hermitian Matrix Computations

Block Coordinate Descent Methods for Optimization under J-Orthogonality Constraints with Applications

An Ulm-like Cayley Transform Method for Inverse Eigenvalue Problems with Multiple Eigenvalues

A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints

Generalized Golub-Kahan bidiagonalization for nonsymmetric saddle point systems

An Inverse Extreme Eigenpair Problem and Its Parallel Iterative Solution for Symmetric Tridiagonal Matrices

A block Lanczos method for large-scale quadratic minimization problems with orthogonality constraints

The Joint Bidiagonalization Process with Partial Reorthogonalization

Composite orthogonal projection methods for large matrix eigenproblems