Abstract:This paper considers the minimization of a sum of an expectation-valued smooth nonconvex function and a nonsmooth block-separable convex regularizer. By combining a randomized block-coordinate descent method with a proximal variable sample-size stochastic gradient (VSSG) method, we propose a randomized block proximal VSSG algorithm. In each iteration, a single block is randomly chosen to updates its estimates by {a VSSG scheme} with an increasing batch of sampled gradients, while the remaining blocks are kept invariant. By appropriately chosen batch sizes, we prove that every limit point for almost every sample path is a stationary point when blocks are chosen either randomly or cyclically. We further show that the ergodic mean-squared error of the gradient mapping {diminishes at the rate of $\mathcal{O}(1/K) $ where $K$denotes the iteration index} and establish that the iteration and oracle complexity to obtain an $\epsilon$-stationary point are $\mathcal{O}(1/\epsilon )$ and $\mathcal{O}(1/\epsilon^2)$, respectively. Furthermore, under a $ {\mu}$-proximal Polyak-{\L}ojasiewicz condition with the batch size increasing at a suitable geometric rate, we prove that the suboptimality diminishes at a {\em geometric} rate, the {\em optimal} deterministic rate. In addition, if $L_{\rm ave}$ denotes the average of block-specific Lipschitz constants, the iteration and oracle complexity to obtain an $\epsilon$-optimal solution are $\mathcal{O}( {(L_{\rm ave}/\mu)}\ln(1/\epsilon))$ and $\mathcal{O}\left( (1/\epsilon)^{1+c} \right)$, respectively, {matching} the deterministic result. When $n=1$, we obtainthe {\em optimal} \red{oracle complexity bound} $\mathcal{O}(1/\epsilon) $ while $c>0$ when $n\geq 2$ represents the positive cost of multiple blocks. Finally, preliminary numerical experiments support our theoretical findings.

Mini-batch stochastic subgradient for functional constrained optimization

Stochastic subgradient for composite optimization with functional constraints

Random minibatch subgradient algorithms for convex problems with functional constraints

Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization

Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models

Mini-batch stochastic Nesterov's smoothing method for constrained convex stochastic composite optimization

Nonsmooth Projection-Free Optimization with Functional Constraints

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

A Proximal-Gradient Method for Constrained Optimization

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization.

Mini-batch stochastic gradient descent with dynamic sample sizes

A Unified Analysis for the Subgradient Methods Minimizing Composite Nonconvex, Nonsmooth and Non-Lipschitz Functions.

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

A Randomized Block Proximal Variable Sample-size Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization