Abstract:Despite the success of stochastic variance-reduced gradient (SVRG) algorithms in solving large-scale problems, their stochastic gradient complexity often scales linearly with data size and is expensive for huge data. Accordingly, we propose a hybrid stochastic-deterministic minibatch proximal gradient ( HSDMPG ) algorithm for strongly convex problems with linear prediction structure, e.g., least squares and logistic/softmax regression. HSDMPG enjoys improved computational complexity that is data-size-independent for large-scale problems. It iteratively samples an evolving minibatch of individual losses to estimate the original problem, and can efficiently minimize the sampled subproblems. For strongly convex loss of $n$ components, HSDMPG attains an $\epsilon$ -optimization-error within $\mathcal {O} \left(\kappa \log ^{\zeta +1}\left(\frac{1}{\epsilon }\right)\frac{1}{\epsilon }\bigwedge n\log ^{\zeta }\left(\frac{1}{\epsilon }\right)\right)$ stochastic gradient evaluations, where $\kappa$ is condition number, $\zeta =1$ for quadratic loss and $\zeta =2$ for generic loss. For large-scale problems, our complexity outperforms those of SVRG-type algorithms with/without dependence on data size. Particularly, when $\epsilon =\mathcal {O}(1/\sqrt{n})$ which matches the intrinsic excess error of a learning model and is sufficient for generalization, our complexity for quadratic and generic losses is respectively $\mathcal {O} (n^{0.5}\log ^{2}(n))$ and $\mathcal {O} (n^{0.5}\log ^{3}(n))$ , which for the first time achieves optimal generalization in less than a single pass over data. Besides, we extend HSDMPG to online strongly convex problems and prove its higher efficiency over the prior algorithms. Numerical results demonstrate the computational advantages of HSDMPG .

On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization.

On the Iteration Complexity Analysis of Stochastic Primal-Dual Hybrid Gradient Approach with High Probability

A Distributed Stochastic Proximal-Gradient Algorithm for Composite Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Smoothing Stochastic Gradient Method for Composite Optimization

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

Stochastic Primal-Dual Algorithms with Faster Convergence Than O(1/√T) for Problems Without Bilinear Structure

Stochastic Primal-Dual Algorithms with Faster Convergence than $O(1/\sqrt{T})$ for Problems without Bilinear Structure

An Edge-based Stochastic Proximal Gradient Algorithm for Decentralized Composite Optimization

Stochastic Primal Dual Fixed Point Method for Composite Optimization

A Stochastic Primal-Dual Algorithm for Composite Constrained Optimization

A Hybrid Stochastic-Deterministic Minibatch Proximal Gradient Method for Efficient Optimization and Generalization.

Precompact Convergence of the Nonconvex Primal–Dual Hybrid Gradient Algorithm

Stochastic Primal–Dual Hybrid Gradient Algorithm with Adaptive Step Sizes

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

Damping Proximal Coordinate Descent Algorithm for Non-Convex Regularization

Stochastic Primal Dual Hybrid Gradient Algorithm with Adaptive Step-Sizes

A Regularized Newton Method for \({\boldsymbol{\ell}}_{q}\) -Norm Composite Optimization Problems.

Primal Dual Alternating Proximal Gradient Algorithms for Nonsmooth Nonconvex Minimax Problems with Coupled Linear Constraints

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

A proximal stochastic gradient method with progressive variance reduction