Abstract:This paper introduces a novel algorithm to approximate the matrix with minimum nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood as the convex relaxation of a rank minimization problem and arises in many important applications as in the task of recovering a large matrix from a small subset of its entries (the famous Netflix problem). Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries. This paper develops a simple first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in which the optimal solution has low rank. The algorithm is iterative, produces a sequence of matrices $\{\boldsymbol{X}^k,\boldsymbol{Y}^k\}$, and at each step mainly performs a soft-thresholding operation on the singular values of the matrix $\boldsymbol{Y}^k$. There are two remarkable features making this attractive for low-rank matrix completion problems. The first is that the soft-thresholding operation is applied to a sparse matrix; the second is that the rank of the iterates $\{\boldsymbol{X}^k\}$ is empirically nondecreasing. Both these facts allow the algorithm to make use of very minimal storage space and keep the computational cost of each iteration low. On the theoretical side, we provide a convergence analysis showing that the sequence of iterates converges. On the practical side, we provide numerical examples in which $1,000\times1,000$ matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate that our approach is amenable to very large scale problems by recovering matrices of rank about 10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are connected with the recent literature on linearized Bregman iterations for $\ell_1$ minimization, and we develop a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms.

An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Fast and Accurate Matrix Completion Via Truncated Nuclear Norm Regularization

A generalized Nystrom method with column sketching for low-rank approximation of nonsymmetric matrices

Extended Gauss-Newton and ADMM-Gauss-Newton Algorithms for Low-Rank Matrix Optimization

New algorithms to compute the nearness symmetric solution of the matrix equation

A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

Approximation Conjugate Gradient Method for Low-Rank Matrix Recovery

Low rank approximation of the symmetric positive semidefinite matrix

Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

On the Unreasonable Effectiveness of Single Vector Krylov Methods for Low-Rank Approximation

Multiplicative iterative algorithm for the low rank approximation of the symmetric positive semi-definite matrix

An Accelerated Randomized Kaczmarz Method Via Low-Rank Approximation

Making the Nyström method highly accurate for low-rank approximations

Fast Gradient Method for Low-Rank Matrix Estimation

A Singular Value Thresholding Algorithm for Matrix Completion

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

A sketch-and-project method for solving the matrix equation AXB = C

Preconditioned Low-Rank Riemannian Optimization for Symmetric Positive Definite Linear Matrix Equations

Randomized Algorithms for Symmetric Nonnegative Matrix Factorization

An algorithm for low-rank matrix factorization and its applications

Efficient Randomized Algorithms for the Fixed-Precision Low-Rank Matrix Approximation