Abstract:This paper considers the minimization of a sum of an expectation-valued smooth nonconvex function and a nonsmooth block-separable convex regularizer. By combining a randomized block-coordinate descent method with a proximal variable sample-size stochastic gradient (VSSG) method, we propose a randomized block proximal VSSG algorithm. In each iteration, a single block is randomly chosen to updates its estimates by {a VSSG scheme} with an increasing batch of sampled gradients, while the remaining blocks are kept invariant. By appropriately chosen batch sizes, we prove that every limit point for almost every sample path is a stationary point when blocks are chosen either randomly or cyclically. We further show that the ergodic mean-squared error of the gradient mapping {diminishes at the rate of $\mathcal{O}(1/K) $ where $K$denotes the iteration index} and establish that the iteration and oracle complexity to obtain an $\epsilon$-stationary point are $\mathcal{O}(1/\epsilon )$ and $\mathcal{O}(1/\epsilon^2)$, respectively. Furthermore, under a $ {\mu}$-proximal Polyak-{\L}ojasiewicz condition with the batch size increasing at a suitable geometric rate, we prove that the suboptimality diminishes at a {\em geometric} rate, the {\em optimal} deterministic rate. In addition, if $L_{\rm ave}$ denotes the average of block-specific Lipschitz constants, the iteration and oracle complexity to obtain an $\epsilon$-optimal solution are $\mathcal{O}( {(L_{\rm ave}/\mu)}\ln(1/\epsilon))$ and $\mathcal{O}\left( (1/\epsilon)^{1+c} \right)$, respectively, {matching} the deterministic result. When $n=1$, we obtainthe {\em optimal} \red{oracle complexity bound} $\mathcal{O}(1/\epsilon) $ while $c>0$ when $n\geq 2$ represents the positive cost of multiple blocks. Finally, preliminary numerical experiments support our theoretical findings.

Stochastic optimization with importance sampling for regularized loss minimization

Adaptive Importance Sampling for Efficient Stochastic Root Finding and Quantile Estimation

Low-Cost Lipschitz-Independent Adaptive Importance Sampling of Stochastic Gradients.

Importance Sampling for Minimization of Tail Risks: A Tutorial

Multiple importance sampling for stochastic gradient estimation

ON EFFICIENCIES OF STOCHASTIC OPTIMIZATION PROCEDURES UNDER IMPORTANCE SAMPLING

Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method

A Proximal Stochastic Quasi-Newton Algorithm with Dynamical Sampling and Stochastic Line Search

Local smoothness in variance reduced optimization

Importance Sampling for Stochastic Gradient Descent in Deep Neural Networks

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Adaptive Coordinate Sampling for Stochastic Primal-Dual Optimization.

A Randomized Block Proximal Variable Sample-size Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization

Optimization with Discrete Simultaneous Perturbation Stochastic Approximation Using Noisy Loss Function Measurements

Importance Sampled Stochastic Optimization for Variational Inference

Stochastic alternating structure-adapted proximal gradient descent method with variance reduction for nonconvex nonsmooth optimization

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Stochastic Optimization with Decision-Dependent Distributions

Subsampled Optimization: Statistical Guarantees, Mean Squared Error Approximation, and Sampling Method

A Stochastic Approximation Framework for a Class of Randomized Optimization Algorithms

Optimal sampling for stochastic and natural gradient descent