Abstract:Uncertainty sampling is a prevalent active learning algorithm that queries sequentially the annotations of data samples which the current prediction model is uncertain about. However, the usage of uncertainty sampling has been largely heuristic: (i) There is no consensus on the proper definition of "uncertainty" for a specific task under a specific loss; (ii) There is no theoretical guarantee that prescribes a standard protocol to implement the algorithm, for example, how to handle the sequentially arrived annotated data under the framework of optimization algorithms such as stochastic gradient descent. In this work, we systematically examine uncertainty sampling algorithms under both stream-based and pool-based active learning. We propose a notion of equivalent loss which depends on the used uncertainty measure and the original loss function and establish that an uncertainty sampling algorithm essentially optimizes against such an equivalent loss. The perspective verifies the properness of existing uncertainty measures from two aspects: surrogate property and loss convexity. Furthermore, we propose a new notion for designing uncertainty measures called \textit{loss as uncertainty}. The idea is to use the conditional expected loss given the features as the uncertainty measure. Such an uncertainty measure has nice analytical properties and generality to cover both classification and regression problems, which enable us to provide the first generalization bound for uncertainty sampling algorithms under both stream-based and pool-based settings, in the full generality of the underlying model and problem. Lastly, we establish connections between certain variants of the uncertainty sampling algorithms with risk-sensitive objectives and distributional robustness, which can partly explain the advantage of uncertainty sampling algorithms when the sample size is small.

A method to find an efficient and robust sampling strategy under model uncertainty

Comparative study of metamodeling and sampling design for expensive and semi-expensive simulation models under uncertainty

Improved sampling strategies for ensemble-based optimization

A Sequential Sampling Approach for Multi-Fidelity Surrogate Modeling-Based Robust Design Optimization

Bayesian Uncertainty Estimation Under Complex Sampling

Efficient metamodel-based importance sampling coupled with single-loop estimation method for parameter global reliability sensitivity analysis

Distributionally Robust Stratified Sampling for Stochastic Simulations with Multiple Uncertain Input Models

Robust Data-Driven Decisions Under Model Uncertainty

Understanding Uncertainty Sampling

Efficient Importance Sampling for Rare Event Simulation with Applications

A New Statistical Approach for the Analysis of Uncertain Systems

Iterative importance sampling with Markov chain Monte Carlo sampling in robust Bayesian analysis

A reinforced learning approach to optimal design under model uncertainty

A Statistical Perspective on Linear Programs with Uncertain Parameters

Robust Stochastic Optimization with Rare-Event Modeling

Portfolio selection under model uncertainty: a penalized moment-based optimization approach

A General Framework for Importance Sampling with Latent Markov Processes

Probability Sampling Designs: Principles for Choice of Design and Balancing

Dynamic Sampling Allocation and Design Selection.

A Statistical Theory for the Analysis of Uncertain Systems

Sequential Sampling Framework for Metamodeling Uncertainty Reduction in Multilevel Optimization of Hierarchical Systems