Abstract:In this paper we consider finite sum composite convex optimization problems with many functional constraints. The objective function is expressed as a finite sum of two terms, one of which admits easy computation of (sub)gradients while the other is amenable to proximal evaluations. We assume a generalized bounded gradient condition on the objective which allows us to simultaneously tackle both smooth and nonsmooth problems. We also consider the cases of both with and without a strong convexity property. Further, we assume that each constraint set is given as the level set of a convex but not necessarily differentiable function. We reformulate the constrained finite sum problem into a stochastic optimization problem for which the stochastic subgradient projection method from [17] specializes to a collection of mini-batch variants, with different mini-batch sizes for the objective function and functional constraints, respectively. More specifically, at each iteration, our algorithm takes a mini-batch stochastic proximal subgradient step aimed at minimizing the objective function and then a subsequent mini-batch subgradient projection step minimizing the feasibility violation. By specializing different mini-batching strategies, we derive exact expressions for the stepsizes as a function of the mini-batch size and in some cases we also derive insightful stepsize-switching rules which describe when one should switch from a constant to a decreasing stepsize regime. We also prove sublinear convergence rates for the mini-batch subgradient projection algorithm which depend explicitly on the mini-batch sizes and on the properties of the objective function. Numerical results also show a better performance of our mini-batch scheme over its single-batch counterpart.

Mini-batch stochastic gradient descent with dynamic sample sizes

Mini-batch stochastic subgradient for functional constrained optimization

Statistical Analysis of Fixed Mini-Batch Gradient Descent Estimator

Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models

Accelerating Stochastic Gradient Descent Using Antithetic Sampling.

Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization

Dynamic Batch Adaptation

Accelerating Minibatch Stochastic Gradient Descent Using Typicality Sampling

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization

A stochastic gradient method with variance control and variable learning rate for Deep Learning

Never Go Full Batch (in Stochastic Convex Optimization)

Aiming towards the minimizers: fast convergence of SGD for overparametrized problems

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Discrete error dynamics of mini-batch gradient descent for least squares regression

Locally Optimal Descent for Dynamic Stepsize Scheduling

Shuffling Gradient Descent-Ascent with Variance Reduction for Nonconvex-Strongly Concave Smooth Minimax Problems

Diffusion Stochastic Optimization for Min-Max Problems

Carathéodory Sampling for Stochastic Gradient Descent

Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems

Optimal sampling for stochastic and natural gradient descent