Abstract:Uncertainty sampling is a prevalent active learning algorithm that queries sequentially the annotations of data samples which the current prediction model is uncertain about. However, the usage of uncertainty sampling has been largely heuristic: (i) There is no consensus on the proper definition of "uncertainty" for a specific task under a specific loss; (ii) There is no theoretical guarantee that prescribes a standard protocol to implement the algorithm, for example, how to handle the sequentially arrived annotated data under the framework of optimization algorithms such as stochastic gradient descent. In this work, we systematically examine uncertainty sampling algorithms under both stream-based and pool-based active learning. We propose a notion of equivalent loss which depends on the used uncertainty measure and the original loss function and establish that an uncertainty sampling algorithm essentially optimizes against such an equivalent loss. The perspective verifies the properness of existing uncertainty measures from two aspects: surrogate property and loss convexity. Furthermore, we propose a new notion for designing uncertainty measures called \textit{loss as uncertainty}. The idea is to use the conditional expected loss given the features as the uncertainty measure. Such an uncertainty measure has nice analytical properties and generality to cover both classification and regression problems, which enable us to provide the first generalization bound for uncertainty sampling algorithms under both stream-based and pool-based settings, in the full generality of the underlying model and problem. Lastly, we establish connections between certain variants of the uncertainty sampling algorithms with risk-sensitive objectives and distributional robustness, which can partly explain the advantage of uncertainty sampling algorithms when the sample size is small.

Active Learning with Sampling by Uncertainty and Density for Word Sense Disambiguation and Text Classification.

Active Learning With Sampling by Uncertainty and Density for Data Annotations

Uncertainty Sampling Based Active Learning with Diversity Constraint by Sparse Selection.

Uncertainty-Based Active Learning Via Sparse Modeling for Image Classification

Uncertainty-aware Complementary Label Queries for Active Learning

Learning Representative Examples for Data Annotation

A Density-Based Re-ranking Technique for Active Learning for Data Annotations

Learning Distinctive Margin Toward Active Domain Adaptation

Active Sentence Learning by Adversarial Uncertainty Sampling in Discrete Space

Multi-Class Active Learning by Uncertainty Sampling with Diversity Maximization

Bidirectional Uncertainty-Based Active Learning for Open Set Annotation

Active Learning Via Sequential Design and Uncertainty Sampling

Active Learning for Chinese Word Sense Disambiguation

Active learning with sampling by joint global-local uncertainty for salient object detection

Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning

Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation

Understanding Uncertainty Sampling

Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training

Improving Uncertainty Sampling with Bell Curve Weight Function

Uncertainty for Active Learning on Graphs

A Serial Sample Selection Framework for Active Learning.