Abstract:We consider a class of stochastic smooth convex optimization problems under rather general assumptions on the noise in the stochastic gradient observation. As opposed to the classical problem setting in which the variance of noise is assumed to be uniformly bounded, herein we assume that the variance of stochastic gradients is related to the "sub-optimality" of the approximate solutions delivered by the algorithm. Such problems naturally arise in a variety of applications, in particular, in the well-known generalized linear regression problem in statistics. However, to the best of our knowledge, none of the existing stochastic approximation algorithms for solving this class of problems attain optimality in terms of the dependence on accuracy, problem parameters, and mini-batch size. We discuss two non-Euclidean accelerated stochastic approximation routines--stochastic accelerated gradient descent (SAGD) and stochastic gradient extrapolation (SGE)--which carry a particular duality relationship. We show that both SAGD and SGE, under appropriate conditions, achieve the optimal convergence rate, attaining the optimal iteration and sample complexities simultaneously. However, corresponding assumptions for the SGE algorithm are more general; they allow, for instance, for efficient application of the SGE to statistical estimation problems under heavy tail noises and discontinuous score functions. We also discuss the application of the SGE to problems satisfying quadratic growth conditions, and show how it can be used to recover sparse solutions. Finally, we report on some simulation experiments to illustrate numerical performance of our proposed algorithms in high-dimensional settings.

Scalable Statistical Inference in Non-parametric Least Squares

Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Data

A Statistical Online Inference Approach in Averaged Stochastic Approximation

Statistical inference for model parameters in stochastic gradient descent

Online Statistical Inference for Gradient-free Stochastic Optimization

Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regression

A New Reliability Method Combining Adaptive Kriging and Active Variance Reduction Using Multiple Importance Sampling

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning

Online Statistical Inference for Stochastic Optimization via Gradient-free Kiefer-Wolfowitz Methods

Asymptotic Analysis of Sampling Estimators for Randomized Numerical Linear Algebra Algorithms

Accelerated stochastic approximation with state-dependent noise

A Fully Online Approach for Covariance Matrices Estimation of Stochastic Gradient Descent Solutions

Statistical Inference for Online Decision Making via Stochastic Gradient Descent

Scalable inference in functional linear regression with streaming data

Online Learning under A Separable Stochastic Approximation Framework

Asymptotic Time-Uniform Inference for Parameters in Averaged Stochastic Approximation

Robust Stochastic Approximation Approach to Stochastic Programming

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models.