Abstract:This paper considers general rank-constrained optimization problems that minimize a general objective function ${f}( {X})$ over the set of rectangular ${n}times {m}$ matrices that have rank at most r. To tackle the rank constraint and also to reduce the computational burden, we factorize $ {X}$ into $ {U} {V} ^{mathrm {T}}$ where $ {U}$ and $ {V}$ are ${n}times {r}$ and ${m}times {r}$ matrices, respectively, and then optimize over the small matrices $ {U}$ and $ {V}$ . We characterize the global optimization geometry of the nonconvex factored problem and show that the corresponding objective function satisfies the robust strict saddle property as long as the original objective function f satisfies restricted strong convexity and smoothness properties, ensuring global convergence of many local search algorithms (such as noisy gradient descent) in polynomial time for solving the factored problem. We also provide a comprehensive analysis for the optimization geometry of a matrix factorization problem where we aim to find ${n}times {r}$ and ${m}times {r}$ matrices $ {U}$ and $ {V}$ such that $ {U} {V} ^{mathrm {T}}$ approximates a given matrix $ {X}^star $ . Aside from the robust strict saddle property, we show that the objective function of the matrix factorization problem has no spurious local minima and obeys the strict saddle property not only for the exact-parameterization case where $mathrm {rank}( {X}^star) = {r}$ , but also for the over-parameterization case where $mathrm {rank}( {X}^star) < {r}$ and the under-parameterization case where $mathrm {rank}( {X}^star) > {r}$ . These geometric properties imply that a number of iterative optimization algorithms (such as gradient descent) converge to a global solution with random initialization.

Optimal l-one Rank One Matrix Decompositions

Optimal Low-Rank Matrix Completion: Semidefinite Relaxations and Eigenvector Disjunctions

Computational complexity of decomposing a symmetric matrix as a sum of positive semidefinite and diagonal matrices

A lower bound for the Balan--Jiang matrix problem

General Low-rank Matrix Optimization: Geometric Analysis and Sharper Bounds

Finding approximately rank-one submatrices with the nuclear norm and l1 norm

Matrices with high completely positive semidefinite rank

Optimal $(0,1)$-Matrix Completion with Majorization Ordered Objectives (To the memory of Pravin Varaiya)

Rank Properties and Computational Methods for Orthogonal Tensor Decompositions

Nonnegative sum-symmetric matrices, optimal-score partitions, and optimal resource allocation

Decomposing tensors via rank-one approximations

Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions

On the analysis of optimization with fixed-rank matrices: a quotient geometric view

Numerical Matrix Decomposition

Optimal size of linear matrix inequalities in semidefinite approaches to polynomial optimization

On the geometric interpretation of the nonnegative rank

A Lower-Upper-Lower Block Triangular Decomposition with Minimal Off-Diagonal Ranks

Preconditioned Low-Rank Riemannian Optimization for Symmetric Positive Definite Linear Matrix Equations

Low Rank Tensor Decompositions and Approximations

The Global Optimization Geometry of Low-Rank Matrix Optimization

Decomposition of linear tensor transformations