Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Naive Penalized Spline Estimators of Derivatives Achieve Optimal Rates of Convergence

Estimating Functions and Derivatives Via Adaptive Penalized Splines.

Penalized spline estimation of principal components for sparse functional data: rates of convergence

Adaptive Penalized Splines for Data Smoothing

Improved convergence rates of nonparametric penalized regression under misspecified total variation

Fast Bivariate Penalized Splines: the Sandwich Smoother

The Performance of Cross-Validation and Maximum Likelihood Estimators of Spline Smoothing Parameters

Dynamic Penalized Splines for Streaming Data

Solving Euler equations via two-stage nonparametric penalized splines

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality

Nonparametric Estimation via Partial Derivatives

Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates

Statistica Sinica Preprint No : SS-2020-0167 Title Dynamic Penalized Splines for Streaming Data

Estimating a Function and Its Derivatives Under a Smoothness Condition

Derivatives of Stochastic Gradient Descent in parametric optimization

Sparse and Efficient Estimation for Partial Spline Models with Increasing Dimension

Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Predictive Local Smoothness for Stochastic Gradient Methods

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model