Abstract: Proving algorithm-dependent generalization error bounds for gradient-type optimization methods has attracted significant attention recently in learning theory. However, most existing trajectory-based analyses require either restrictive assumptions on the learning rate (e.g., fast decreasing learning rate), or continuous injected noise (such as the Gaussian noise in Langevin dynamics). In this paper, we introduce a new discrete data-dependent prior to the PAC-Bayesian framework, and prove a high probability generalization bound of order $O(\frac{1}{n}\cdot \sum_{t=1}^T(\gamma_t/\varepsilon_t)^2\left\|{\mathbf{g}_t}\right\|^2)$ for Floored GD (i.e. a version of gradient descent with precision level $\varepsilon_t$), where $n$ is the number of training samples, $\gamma_t$ is the learning rate at step $t$, $\mathbf{g}_t$ is roughly the difference of the gradient computed using all samples and that using only prior samples. $\left\|{\mathbf{g}_t}\right\|$ is upper bounded by and and typical much smaller than the gradient norm $\left\|{\nabla f(W_t)}\right\|$. We remark that our bound holds for nonconvex and nonsmooth scenarios. Moreover, our theoretical results provide numerically favorable upper bounds of testing errors (e.g., $0.037$ on MNIST). Using a similar technique, we can also obtain new generalization bounds for certain variants of SGD. Furthermore, we study the generalization bounds for gradient Langevin Dynamics (GLD). Using the same framework with a carefully constructed continuous prior, we show a new high probability generalization bound of order $O(\frac{1}{n} + \frac{L^2}{n^2}\sum_{t=1}^T(\gamma_t/\sigma_t)^2)$ for GLD. The new $1/n^2$ rate is due to the concentration of the difference between the gradient of training samples and that of the prior.

Error bounds for overdetermined and underdetermined generalized centred simplex gradients

A general framework for floating point error analysis of simplex derivatives

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

The Sample Complexity of Gradient Descent in Stochastic Convex Optimization

Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization

Generalization Bounds for Gradient Methods via Discrete and Continuous Prior

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Sharper Bounds for Proximal Gradient Algorithms with Errors

Curvature Aligned Simplex Gradient: Principled Sample Set Construction For Numerical Differentiation

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Laplacian Smoothing Gradient Descent

The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization

Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions

Subspace Quasi-Newton Method with Gradient Approximation

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One

Global Convergence of Non-Convex Gradient Descent for Computing Matrix Squareroot

Subgradient sampling for nonsmooth nonconvex minimization

Dual Space Preconditioning for Gradient Descent