Abstract:This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete probability distribution $(p_1, \dots, p_n)$, where the probabilities of the output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ of the sampling algorithm must be specified using at most $k$ bits of precision? We present a theoretical framework for formulating this problem and provide new techniques for finding sampling algorithms that are optimal both statistically (in the sense of sampling accuracy) and information-theoretically (in the sense of entropy consumption). We leverage these results to build a system that, for a broad family of measures of statistical accuracy, delivers a sampling algorithm whose expected entropy usage is minimal among those that induce the same distribution (i.e., is "entropy-optimal") and whose output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ is a closest approximation to the target distribution $(p_1, \dots, p_n)$ among all entropy-optimal sampling algorithms that operate within the specified $k$-bit precision. This optimal approximate sampler is also a closer approximation than any (possibly entropy-suboptimal) sampler that consumes a bounded amount of entropy with the specified precision, a class which includes floating-point implementations of inversion sampling and related methods found in many software libraries. We evaluate the accuracy, entropy consumption, precision requirements, and wall-clock runtime of our optimal approximate sampling algorithms on a broad set of distributions, demonstrating the ways that they are superior to existing approximate samplers and establishing that they often consume significantly fewer resources than are needed by exact samplers.

Algorithms for generating small random samples

Algorithms for Generating Small Random Samples

Simple, Optimal Algorithms for Random Sampling Without Replacement

A Novel Quota Sampling Algorithm for Generating Representative Random Samples given Small Sample Size

The Randomness Recycler: A new technique for perfect sampling

Consistent Sampling with Replacement

An asymptotically optimal, online algorithm for weighted random sampling with replacement

Batched ranged random integer generation

Optimal Discrete Uniform Generation from Coin Flips, and Applications

Sampling parts of random integer partitions: a probabilistic and asymptotic analysis

An Introduction to Mrandom 3.0

Large Deviations Inequalities for Unequal Probability Sampling Without Replacement

On the Average Runtime of an Open Source Binomial Random Variate Generation Algorithm

Reducing Randomness Via Irrational Numbers

On-Line Selection of Alternating Subsequences from a Random Sample

A Simple Algorithm for Worst-Case Optimal Join and Sampling

Optimal Approximate Sampling from Discrete Probability Distributions

Detecting random sets by samplings from their values

Variance-Optimal Offline and Streaming Stratified Random Sampling

Relevant Sampling in a reproducing kernel subspace of Orlicz space