Abstract:This paper considers general rank-constrained optimization problems that minimize a general objective function ${f}( {X})$ over the set of rectangular ${n}times {m}$ matrices that have rank at most r. To tackle the rank constraint and also to reduce the computational burden, we factorize $ {X}$ into $ {U} {V} ^{mathrm {T}}$ where $ {U}$ and $ {V}$ are ${n}times {r}$ and ${m}times {r}$ matrices, respectively, and then optimize over the small matrices $ {U}$ and $ {V}$ . We characterize the global optimization geometry of the nonconvex factored problem and show that the corresponding objective function satisfies the robust strict saddle property as long as the original objective function f satisfies restricted strong convexity and smoothness properties, ensuring global convergence of many local search algorithms (such as noisy gradient descent) in polynomial time for solving the factored problem. We also provide a comprehensive analysis for the optimization geometry of a matrix factorization problem where we aim to find ${n}times {r}$ and ${m}times {r}$ matrices $ {U}$ and $ {V}$ such that $ {U} {V} ^{mathrm {T}}$ approximates a given matrix $ {X}^star $ . Aside from the robust strict saddle property, we show that the objective function of the matrix factorization problem has no spurious local minima and obeys the strict saddle property not only for the exact-parameterization case where $mathrm {rank}( {X}^star) = {r}$ , but also for the over-parameterization case where $mathrm {rank}( {X}^star) < {r}$ and the under-parameterization case where $mathrm {rank}( {X}^star) > {r}$ . These geometric properties imply that a number of iterative optimization algorithms (such as gradient descent) converge to a global solution with random initialization.

Computing Bouligand stationary points efficiently in low-rank optimization

Low-rank optimization methods based on projected-projected gradient descent that accumulate at Bouligand stationary points

Projected gradient descent accumulates at Bouligand stationary points

Optimization over bounded-rank matrices through a desingularization enables joint global and local guarantees

A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

General Low-rank Matrix Optimization: Geometric Analysis and Sharper Bounds

A note on stationarity in constrained optimization

On the analysis of optimization with fixed-rank matrices: a quotient geometric view

First-order optimization on stratified sets

Spurious local minima in nonconvex sum-of-squares optimization

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization

Optimization without Retraction on the Random Generalized Stiefel Manifold

Bures-Wasserstein minimizing geodesics between covariance matrices of different ranks

The Computational Complexity of Finding Stationary Points in Non-Convex Optimization

The Global Optimization Geometry of Low-Rank Matrix Optimization

Large-Scale Convex Minimization with a Low-Rank Constraint

Goldstein Stationarity in Lipschitz Constrained Optimization

Guarantees of Riemannian Optimization for Low Rank Matrix Recovery

Preconditioned Low-Rank Riemannian Optimization for Symmetric Positive Definite Linear Matrix Equations

A Riemannian optimization method to compute the nearest singular pencil