Abstract:Composite convex optimization problems which include both a nonsmooth term and a low-rank promoting term have important applications in machine learning and signal processing, such as when one wishes to recover an unknown matrix that is simultaneously low-rank and sparse. However, such problems are highly challenging to solve in large-scale: the low-rank promoting term prohibits efficient implementations of proximal methods for composite optimization and even simple subgradient methods. On the other hand, methods which are tailored for low-rank optimization, such as conditional gradient-type methods, which are often applied to a smooth approximation of the nonsmooth objective, are slow since their runtime scales with both the large Lipshitz parameter of the smoothed gradient vector and with $1/\epsilon$. In this paper we develop efficient algorithms for \textit{stochastic} optimization of a strongly-convex objective which includes both a nonsmooth term and a low-rank promoting term. In particular, to the best of our knowledge, we present the first algorithm that enjoys all following critical properties for large-scale problems: i) (nearly) optimal sample complexity, ii) each iteration requires only a single \textit{low-rank} SVD computation, and iii) overall number of thin-SVD computations scales only with $\log{1/\epsilon}$ (as opposed to $\textrm{poly}(1/\epsilon)$ in previous methods). We also give an algorithm for the closely-related finite-sum setting. At the heart of our results lie a novel combination of a variance-reduction technique and the use of a \textit{weak-proximal oracle} which is key to obtaining all above three properties simultaneously.

Fast Convergence to Second-Order Stationary Point through Random Subspace Optimization

Randomized subspace gradient method for constrained optimization

Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimization

Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

Randomized subspace regularized Newton method for unconstrained non-convex optimization

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Convergence to a second-order critical point by a primal-dual interior point trust-region method for nonlinear semidefinite programming

Convergence and worst-case complexity of adaptive Riemannian trust-region methods for optimization on manifolds

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

Stochastic Trust-Region Algorithm in Random Subspaces with Convergence and Expected Complexity Analyses

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Gradient Norm Regularization Second-Order Algorithms for Solving Nonconvex-Strongly Concave Minimax Problems

A Two-Stage Subspace Trust Region Approach for Deep Neural Network Training

Optimal Randomized First-Order Methods for Least-Squares Problems

A Second-Order Cone Based Approach for Solving the Trust Region Subproblem and Its Variants

Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

A Riemannian Subspace Limited-Memory SR1 Trust Region Method

Trust-Region Stochastic Optimization with Variance Reduction Technique