Abstract:This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete probability distribution $(p_1, \dots, p_n)$, where the probabilities of the output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ of the sampling algorithm must be specified using at most $k$ bits of precision? We present a theoretical framework for formulating this problem and provide new techniques for finding sampling algorithms that are optimal both statistically (in the sense of sampling accuracy) and information-theoretically (in the sense of entropy consumption). We leverage these results to build a system that, for a broad family of measures of statistical accuracy, delivers a sampling algorithm whose expected entropy usage is minimal among those that induce the same distribution (i.e., is "entropy-optimal") and whose output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ is a closest approximation to the target distribution $(p_1, \dots, p_n)$ among all entropy-optimal sampling algorithms that operate within the specified $k$-bit precision. This optimal approximate sampler is also a closer approximation than any (possibly entropy-suboptimal) sampler that consumes a bounded amount of entropy with the specified precision, a class which includes floating-point implementations of inversion sampling and related methods found in many software libraries. We evaluate the accuracy, entropy consumption, precision requirements, and wall-clock runtime of our optimal approximate sampling algorithms on a broad set of distributions, demonstrating the ways that they are superior to existing approximate samplers and establishing that they often consume significantly fewer resources than are needed by exact samplers.

Sampling Exactly from the Normal Distribution

An improved exact sampling algorithm for the standard normal distribution

Generation of normal distributions revisited

On normal realizations of discrete-time systems with consideration of finite precision implementation

Optimal Approximate Sampling from Discrete Probability Distributions

A fast normal random number generator

IID Sampling from Intractable Distributions

Nonnormal small jump approximation of infinitely divisible distributions

High-Performance Constant-Time Discrete Gaussian Sampling

The Randomness Recycler: A new technique for perfect sampling

Direct Methods for Exact Truncated Seyuential Tests of the Mean of a Normal Distribution

Sampling normal distribution restricted on multiple regions

Random points are good for universal discretization

Reliable simulation of extremely-truncated log-concave distributions

Computable Absolutely Normal Numbers and Discrepancies

Consistent Sampling with Replacement

The Modified-Half-Normal distribution: Properties and an efficient sampling scheme

Quantum Algorithms for Sampling Log-Concave Distributions and Estimating Normalizing Constants.

Simple, Optimal Algorithms for Random Sampling Without Replacement

A Novel Quota Sampling Algorithm for Generating Representative Random Samples given Small Sample Size