Abstract:Despite the success of stochastic variance-reduced gradient (SVRG) algorithms in solving large-scale problems, their stochastic gradient complexity often scales linearly with data size and is expensive for huge data. Accordingly, we propose a hybrid stochastic-deterministic minibatch proximal gradient ( HSDMPG ) algorithm for strongly convex problems with linear prediction structure, e.g., least squares and logistic/softmax regression. HSDMPG enjoys improved computational complexity that is data-size-independent for large-scale problems. It iteratively samples an evolving minibatch of individual losses to estimate the original problem, and can efficiently minimize the sampled subproblems. For strongly convex loss of $n$ components, HSDMPG attains an $\epsilon$ -optimization-error within $\mathcal {O} \left(\kappa \log ^{\zeta +1}\left(\frac{1}{\epsilon }\right)\frac{1}{\epsilon }\bigwedge n\log ^{\zeta }\left(\frac{1}{\epsilon }\right)\right)$ stochastic gradient evaluations, where $\kappa$ is condition number, $\zeta =1$ for quadratic loss and $\zeta =2$ for generic loss. For large-scale problems, our complexity outperforms those of SVRG-type algorithms with/without dependence on data size. Particularly, when $\epsilon =\mathcal {O}(1/\sqrt{n})$ which matches the intrinsic excess error of a learning model and is sufficient for generalization, our complexity for quadratic and generic losses is respectively $\mathcal {O} (n^{0.5}\log ^{2}(n))$ and $\mathcal {O} (n^{0.5}\log ^{3}(n))$ , which for the first time achieves optimal generalization in less than a single pass over data. Besides, we extend HSDMPG to online strongly convex problems and prove its higher efficiency over the prior algorithms. Numerical results demonstrate the computational advantages of HSDMPG .

Towards Memory-Friendly Deterministic Incremental Gradient Method.

An Incremental Updating Method for Support Vector Machines

An Adaptive Incremental Gradient Method With Support for Non-Euclidean Norms

Gradient Methods with Memory for Minimizing Composite Functions

An optimal randomized incremental gradient method

An inexact variable metric variant of incremental aggregated Forward-Backward method for the large-scale composite optimization problem

Linear Speedup of Incremental Aggregated Gradient Methods on Streaming Data

Incremental Gauss-Newton Descent for Machine Learning

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Compositional Stochastic Average Gradient for Machine Learning and Related Applications

SVGD: A Virtual Gradients Descent Method for Stochastic Optimization

Coresets for Data-efficient Training of Machine Learning Models

Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization

Parallel Asynchronous Stochastic Variance Reduction for Nonconvex Optimization

Shuffling Gradient Descent-Ascent with Variance Reduction for Nonconvex-Strongly Concave Smooth Minimax Problems

Adaptive Variance Reducing for Stochastic Gradient Descent.

Memory gradient method for multiobjective optimization

New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity.

On Convergence of Incremental Gradient for Non-Convex Smooth Functions

Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction

A Hybrid Stochastic-Deterministic Minibatch Proximal Gradient Method for Efficient Optimization and Generalization.