Abstract:Algorithmic stability is an important notion that has proven powerful for deriving generalization bounds for practical algorithms. The last decade has witnessed an increasing number of stability bounds for different algorithms applied on different classes of loss functions. While these bounds have illuminated various properties of optimization algorithms, the analysis of each case typically required a different proof technique with significantly different mathematical tools. In this study, we make a novel connection between learning theory and applied probability and introduce a unified guideline for proving Wasserstein stability bounds for stochastic optimization algorithms. We illustrate our approach on stochastic gradient descent (SGD) and we obtain time-uniform stability bounds (i.e., the bound does not increase with the number of iterations) for strongly convex losses and non-convex losses with additive noise, where we recover similar results to the prior art or extend them to more general cases by using a single proof technique. Our approach is flexible and can be generalizable to other popular optimizers, as it mainly requires developing Lyapunov functions, which are often readily available in the literature. It also illustrates that ergodicity is an important component for obtaining time-uniform bounds -- which might not be achieved for convex or non-convex losses unless additional noise is injected to the iterates. Finally, we slightly stretch our analysis technique and prove time-uniform bounds for SGD under convex and non-convex losses (without additional additive noise), which, to our knowledge, is novel.

Stability-Based Stopping Criterion for Active Learning

Stopping Criterion for Active Learning with Model Stability.

Stopping Criterion for Active Learning Based on Error Stability

Analysis of Stopping Active Learning based on Stabilizing Predictions

A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping

Confidence-based stopping criteria for active learning for data annotation

Uncertainty Sampling Based Active Learning with Diversity Constraint by Sparse Selection.

Uncertainty-Based Active Learning Via Sparse Modeling for Image Classification

Active learning with adaptive regularization

Active Learning for Classification with Maximum Model Change

Maximizing Expected Model Change for Active Learning in Regression

Learning Distinctive Margin Toward Active Domain Adaptation

A Stability Principle for Learning under Non-Stationarity

Deep Active Learning with Noise Stability

Late Stopping: Avoiding Confidently Learning from Mislabeled Examples

Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent

New Balanced Active Learning Model and Optimization Algorithm.

Neural Active Learning on Heteroskedastic Distributions

Stacking and stability

The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification

Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent