Abstract:This paper considers general rank-constrained optimization problems that minimize a general objective function ${f}( {X})$ over the set of rectangular ${n}times {m}$ matrices that have rank at most r. To tackle the rank constraint and also to reduce the computational burden, we factorize $ {X}$ into $ {U} {V} ^{mathrm {T}}$ where $ {U}$ and $ {V}$ are ${n}times {r}$ and ${m}times {r}$ matrices, respectively, and then optimize over the small matrices $ {U}$ and $ {V}$ . We characterize the global optimization geometry of the nonconvex factored problem and show that the corresponding objective function satisfies the robust strict saddle property as long as the original objective function f satisfies restricted strong convexity and smoothness properties, ensuring global convergence of many local search algorithms (such as noisy gradient descent) in polynomial time for solving the factored problem. We also provide a comprehensive analysis for the optimization geometry of a matrix factorization problem where we aim to find ${n}times {r}$ and ${m}times {r}$ matrices $ {U}$ and $ {V}$ such that $ {U} {V} ^{mathrm {T}}$ approximates a given matrix $ {X}^star $ . Aside from the robust strict saddle property, we show that the objective function of the matrix factorization problem has no spurious local minima and obeys the strict saddle property not only for the exact-parameterization case where $mathrm {rank}( {X}^star) = {r}$ , but also for the over-parameterization case where $mathrm {rank}( {X}^star) < {r}$ and the under-parameterization case where $mathrm {rank}( {X}^star) > {r}$ . These geometric properties imply that a number of iterative optimization algorithms (such as gradient descent) converge to a global solution with random initialization.

A Global Optimization Algorithm for Sparse Mixed Membership Matrix Factorization

Exponentially Convergent Algorithms for Supervised Matrix Factorization

Sparse Nonnegative Matrix Factorization with Genetic Algorithms for Microarray Analysis

Fast randomized algorithms for low-rank matrix approximations with applications in global comparative analysis of a class of data sets

Combining Sparse Approximate Factorizations with Mixed-precision Iterative Refinement

Accelerated Structured Matrix Factorization

Subspace-Constrained Quadratic Matrix Factorization: Algorithm and Applications

A Generalized Latent Factor Model Approach to Mixed-data Matrix Completion with Entrywise Consistency

Factor-Bounded Nonnegative Matrix Factorization

An algorithm for low-rank matrix factorization and its applications

Greedy Orthogonal Pivoting Algorithm for Non-Negative Matrix Factorization.

Robust Semi-nonnegative Matrix Factorization with Adaptive Graph Regularization for Gene Representation

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Understanding the Sparsity: Augmented Matrix Factorization with Sampled Constraints on Unobservables.

A Sparse Algorithm for Non-Negative Matrix Factorization Based on Projection Gradient Method

Factorization Approach for Low-complexity Matrix Completion Problems: Exponential Number of Spurious Solutions and Failure of Gradient Methods

Checking the Sufficiently Scattered Condition using a Global Non-Convex Optimization Software

Robust Matrix Factorization by Majorization Minimization

Stochastic Subsampling for Factorizing Huge Matrices

A Fast Algorithm for Nonnegative Matrix Factorization and Its Convergence.

The Global Optimization Geometry of Low-Rank Matrix Optimization