Abstract:Stochastic model-based methods have received increasing attention lately due to their appealing robustness to the stepsize selection and provable efficiency guarantee. We make two important extensions for improving model-based methods on stochastic weakly convex optimization. First, we propose new minibatch model-based methods by involving a set of samples to approximate the model function in each iteration. For the first time, we show that stochastic algorithms achieve linear speedup over the batch size even for non-smooth and non-convex (particularly, weakly convex) problems. To this end, we develop a novel sensitivity analysis of the proximal mapping involved in each algorithm iteration. Our analysis appears to be of independent interests in more general settings. Second, motivated by the success of momentum stochastic gradient descent, we propose a new stochastic extrapolated model-based method, greatly extending the classic Polyak momentum technique to a wider class of stochastic algorithms for weakly convex optimization. The rate of convergence to some natural stationarity condition is established over a fairly flexible range of extrapolation terms. While mainly focusing on weakly convex optimization, we also extend our work to convex optimization. We apply the minibatch and extrapolated model-based methods to stochastic convex optimization, for which we provide a new complexity bound and promising linear speedup in batch size. Moreover, an accelerated model-based method based on Nesterov's momentum is presented, for which we establish an optimal complexity bound for reaching optimality.

Sharper Analysis for Minibatch Stochastic Proximal Point Methods: Stability, Smoothness, and Deviation.

A Unified Theory of Stochastic Proximal Point Methods without Smoothness

Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum

Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models

Non-asymptotic Analysis of Stochastic Methods for Non-Smooth Non-Convex Regularized Problems

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

The stochastic proximal distance algorithm

New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

Solving Nonsmooth Nonconvex Compound Stochastic Programs with Applications to Risk Measure Minimization

Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimization

Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization

Towards Sharper Risk Bounds for Minimax Problems

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Sharp Analysis of Simple Restarted Stochastic Gradient Methods for Min-Max Optimization

A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai–Borwein Stepsizes

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems