Abstract:This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete probability distribution $(p_1, \dots, p_n)$, where the probabilities of the output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ of the sampling algorithm must be specified using at most $k$ bits of precision? We present a theoretical framework for formulating this problem and provide new techniques for finding sampling algorithms that are optimal both statistically (in the sense of sampling accuracy) and information-theoretically (in the sense of entropy consumption). We leverage these results to build a system that, for a broad family of measures of statistical accuracy, delivers a sampling algorithm whose expected entropy usage is minimal among those that induce the same distribution (i.e., is "entropy-optimal") and whose output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ is a closest approximation to the target distribution $(p_1, \dots, p_n)$ among all entropy-optimal sampling algorithms that operate within the specified $k$-bit precision. This optimal approximate sampler is also a closer approximation than any (possibly entropy-suboptimal) sampler that consumes a bounded amount of entropy with the specified precision, a class which includes floating-point implementations of inversion sampling and related methods found in many software libraries. We evaluate the accuracy, entropy consumption, precision requirements, and wall-clock runtime of our optimal approximate sampling algorithms on a broad set of distributions, demonstrating the ways that they are superior to existing approximate samplers and establishing that they often consume significantly fewer resources than are needed by exact samplers.

Determining the Sampling Size with Maintaining the Probability Distribution.

Importance Sampling of Interval Markov Chains

Optimal Approximate Sampling from Discrete Probability Distributions

sampling with probability matching

Improving Importance Sampling Method in Structural Reliability

Calculation of sample size guaranteeing the required width of the empirical confidence interval with predefined probability

Bayesian Uncertainty Estimation Under Complex Sampling

A Probability Proportional to Size Estimation of a Rare Sensitive Attribute Using a Partial Randomized Response Model with Poisson Distribution

Sampling-Based Approaches to Calculating Marginal Densities

Sampling with replacement vs Poisson sampling: a comparative study in optimal subsampling

Divide, Conquer, Combine Bayesian Decision Tree Sampling

Insufficient Gibbs sampling

An effective likelihood-free approximate computing method with statistical inferential guarantees

On Estimation of Finite Population Proportion

Importance sampling method for Feldman-Cousins confidence intervals

Efficient Importance Sampling for Rare Event Simulation with Applications

Dynamic Sampling Allocation and Design Selection.

Uncertainty Quantification and Confidence Intervals for Naive Rare-Event Estimators

Subsampled Optimization: Statistical Guarantees, Mean Squared Error Approximation, and Sampling Method

Failure Probability Estimation for Black-Box Autonomous Systems using State-Dependent Importance Sampling Proposals

Repeated out of Sample Fusion in the Estimation of Small Tail Probabilities