Abstract:Composite convex optimization problems which include both a nonsmooth term and a low-rank promoting term have important applications in machine learning and signal processing, such as when one wishes to recover an unknown matrix that is simultaneously low-rank and sparse. However, such problems are highly challenging to solve in large-scale: the low-rank promoting term prohibits efficient implementations of proximal methods for composite optimization and even simple subgradient methods. On the other hand, methods which are tailored for low-rank optimization, such as conditional gradient-type methods, which are often applied to a smooth approximation of the nonsmooth objective, are slow since their runtime scales with both the large Lipshitz parameter of the smoothed gradient vector and with $1/\epsilon$. In this paper we develop efficient algorithms for \textit{stochastic} optimization of a strongly-convex objective which includes both a nonsmooth term and a low-rank promoting term. In particular, to the best of our knowledge, we present the first algorithm that enjoys all following critical properties for large-scale problems: i) (nearly) optimal sample complexity, ii) each iteration requires only a single \textit{low-rank} SVD computation, and iii) overall number of thin-SVD computations scales only with $\log{1/\epsilon}$ (as opposed to $\textrm{poly}(1/\epsilon)$ in previous methods). We also give an algorithm for the closely-related finite-sum setting. At the heart of our results lie a novel combination of a variance-reduction technique and the use of a \textit{weak-proximal oracle} which is key to obtaining all above three properties simultaneously.

A Sherman--Morrison--Woodbury approach to solving least squares problems with low-rank updates

Low-Rank Updates of Matrix Square Roots

A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

An algorithm for low-rank matrix factorization and its applications

Krylov Subspace Recycling for Fast Iterative Least-Squares in Machine Learning

Preconditioned Low-Rank Riemannian Optimization for Symmetric Positive Definite Linear Matrix Equations

Low-Rank Matrix Iteration Using Polynomial-Filtered Subspace Extraction.

Smoothed Low Rank and Sparse Matrix Recovery by Iteratively Reweighted Least Squares Minimization

The Sherman–Morrison–Woodbury formula for generalized linear matrix equations and applications

Novel Alternating Update Method for Low Rank Approximation of Structured Matrices

Good and Fast Row-Sparse ah-Symmetric Reflexive Generalized Inverses

Active Subspace: Toward Scalable Low-Rank Learning

On an Iterative Method for Solving the Least Squares Problem of Rank-Deficient Systems

Learning-Based Low-Rank Approximations

Exactly Initialized Recursive Least Squares

Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

Adaptive Low-Rank Methods: Problems on Sobolev Spaces

Gauss-Southwell type descent methods for low-rank matrix optimization

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Faster Algorithms for Schatten-p Low Rank Approximation

Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems