Abstract:We revisit the classical problem of finding an approximately stationary point of the average of $n$ smooth and possibly nonconvex functions. The optimal complexity of stochastic first-order methods in terms of the number of gradient evaluations of individual functions is $\mathcal{O}\left(n + n^{1/2}\varepsilon^{-1}\right)$, attained by the optimal SGD methods $\small\sf\color{green}{SPIDER}$(<a class="link-https" data-arxiv-id="1807.01695" href="https://arxiv.org/abs/1807.01695">arXiv:1807.01695</a>) and $\small\sf\color{green}{PAGE}$(<a class="link-https" data-arxiv-id="2008.10898" href="https://arxiv.org/abs/2008.10898">arXiv:2008.10898</a>), for example, where $\varepsilon$ is the error tolerance. However, i) the big-$\mathcal{O}$ notation hides crucial dependencies on the smoothness constants associated with the functions, and ii) the rates and theory in these methods assume simplistic sampling mechanisms that do not offer any flexibility. In this work we remedy the situation. First, we generalize the $\small\sf\color{green}{PAGE}$ algorithm so that it can provably work with virtually any (unbiased) sampling mechanism. This is particularly useful in federated learning, as it allows us to construct and better understand the impact of various combinations of client and data sampling strategies. Second, our analysis is sharper as we make explicit use of certain novel inequalities that capture the intricate interplay between the smoothness constants and the sampling procedure. Indeed, our analysis is better even for the simple sampling procedure analyzed in the $\small\sf\color{green}{PAGE}$ paper. However, this already improved bound can be further sharpened by a different sampling scheme which we propose. In summary, we provide the most general and most accurate analysis of optimal SGD in the smooth nonconvex regime. Finally, our theoretical findings are supposed with carefully designed experiments.

Automatic and asymptotically optimal data sharpening for nonparametric regression

High-order Data Sharpening with Dependent Errors for Regression Bias Reduction

Data Sharpening Method in Regression Confidence Band

Iterated Data Sharpening

Regularization and variable selection for infinite variance autoregressive models

Adaptive Penalized Splines for Data Smoothing

Nonparametric Variable Selection and Its Application to Additive Models

Restarts subject to approximate sharpness: A parameter-free and optimal scheme for first-order methods

Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functions

Fast stable direct fitting and smoothness selection for generalized additive models

Boosting sharpness-aware training with dynamic neighborhood

Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling

A Tuning-free Robust and Efficient Approach to High-dimensional Regression

An anisotropic variational pansharpening model with adaptive coefficients

An efficient optimization approach for best subset selection in linear regression, with application to model selection and fitting in autoregressive time-series

Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory

Benign overfitting and adaptive nonparametric regression

Consistent Tuning Parameter Selection in High Dimensional Sparse Linear Regression

Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term

How Sharpness-Aware Minimization Minimizes Sharpness?