Abstract:This paper considers the minimization of a sum of an expectation-valued smooth nonconvex function and a nonsmooth block-separable convex regularizer. By combining a randomized block-coordinate descent method with a proximal variable sample-size stochastic gradient (VSSG) method, we propose a randomized block proximal VSSG algorithm. In each iteration, a single block is randomly chosen to updates its estimates by {a VSSG scheme} with an increasing batch of sampled gradients, while the remaining blocks are kept invariant. By appropriately chosen batch sizes, we prove that every limit point for almost every sample path is a stationary point when blocks are chosen either randomly or cyclically. We further show that the ergodic mean-squared error of the gradient mapping {diminishes at the rate of $\mathcal{O}(1/K) $ where $K$denotes the iteration index} and establish that the iteration and oracle complexity to obtain an $\epsilon$-stationary point are $\mathcal{O}(1/\epsilon )$ and $\mathcal{O}(1/\epsilon^2)$, respectively. Furthermore, under a $ {\mu}$-proximal Polyak-{\L}ojasiewicz condition with the batch size increasing at a suitable geometric rate, we prove that the suboptimality diminishes at a {\em geometric} rate, the {\em optimal} deterministic rate. In addition, if $L_{\rm ave}$ denotes the average of block-specific Lipschitz constants, the iteration and oracle complexity to obtain an $\epsilon$-optimal solution are $\mathcal{O}( {(L_{\rm ave}/\mu)}\ln(1/\epsilon))$ and $\mathcal{O}\left( (1/\epsilon)^{1+c} \right)$, respectively, {matching} the deterministic result. When $n=1$, we obtainthe {\em optimal} \red{oracle complexity bound} $\mathcal{O}(1/\epsilon) $ while $c>0$ when $n\geq 2$ represents the positive cost of multiple blocks. Finally, preliminary numerical experiments support our theoretical findings.

Randomized Block Subgradient Methods for Convex Nonsmooth and Stochastic Optimization

Stochastic Block Mirror Descent Methods for Nonsmooth and Stochastic Optimization

Block Stochastic Gradient Iteration for Convex and Nonconvex Optimization.

Stochastic alternating structure-adapted proximal gradient descent method with variance reduction for nonconvex nonsmooth optimization

A Randomized Block Proximal Variable Sample-size Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization

A Randomized Block Proximal Variable Sample-size Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization.

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization

Stochastic Block-Coordinate Gradient Projection Algorithms for Submodular Maximization.

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

A Randomized Block-Coordinate Primal-Dual Method for Large-scale Stochastic Saddle Point Problems

Randomized Block-Coordinate Adaptive Algorithms for Nonconvex Optimization Problems

Asynchronous Variance-Reduced Block Schemes for Composite Non-Convex Stochastic Optimization: Block-Specific Steplengths and Adapted Batch-Sizes.

Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization

Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes

Asynchronous Distributed Optimization via Dual Decomposition and Block Coordinate Subgradient Methods

Asynchronous Stochastic Block Coordinate Descent with Variance Reduction

Randomized Sparse Block Kaczmarz as Randomized Dual Block-Coordinate Descent

Decentralized Gradient-Free Methods for Stochastic Non-Smooth Non-Convex Optimization

Asynchronous Decentralized Accelerated Stochastic Gradient Descent

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Variance-reduced Reshuffling Gradient Descent for Nonconvex Optimization: Centralized and Distributed Algorithms