Abstract:This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete probability distribution $(p_1, \dots, p_n)$, where the probabilities of the output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ of the sampling algorithm must be specified using at most $k$ bits of precision? We present a theoretical framework for formulating this problem and provide new techniques for finding sampling algorithms that are optimal both statistically (in the sense of sampling accuracy) and information-theoretically (in the sense of entropy consumption). We leverage these results to build a system that, for a broad family of measures of statistical accuracy, delivers a sampling algorithm whose expected entropy usage is minimal among those that induce the same distribution (i.e., is "entropy-optimal") and whose output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ is a closest approximation to the target distribution $(p_1, \dots, p_n)$ among all entropy-optimal sampling algorithms that operate within the specified $k$-bit precision. This optimal approximate sampler is also a closer approximation than any (possibly entropy-suboptimal) sampler that consumes a bounded amount of entropy with the specified precision, a class which includes floating-point implementations of inversion sampling and related methods found in many software libraries. We evaluate the accuracy, entropy consumption, precision requirements, and wall-clock runtime of our optimal approximate sampling algorithms on a broad set of distributions, demonstrating the ways that they are superior to existing approximate samplers and establishing that they often consume significantly fewer resources than are needed by exact samplers.

An improved exact sampling algorithm for the standard normal distribution

Sampling Exactly from the Normal Distribution

The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

Event-Based State Estimation: Optimal Algorithm with Generalized Closed Skew Normal Distribution

Central Limit Theorems of a Recursive Stochastic Algorithm with Applications to Adaptive Designs

Efficient and exact sampling of simple graphs with given arbitrary degree sequence

A Berry-Esseen theorem for incomplete U-statistics with Bernoulli sampling

Optimal Approximate Sampling from Discrete Probability Distributions

Efficient Importance Sampling for Rare Event Simulation with Applications

The Modified-Half-Normal distribution: Properties and an efficient sampling scheme

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Nonnormal small jump approximation of infinitely divisible distributions

New algorithms for sampling and diffusion models

Bounds for distributional approximation in the multivariate delta method by Stein's method

A Broad and General Sequential Sampling Scheme

Generation of normal distributions revisited

Asymptotic properties of distributed social sampling algorithm

Achieving Efficiency in Black Box Simulation of Distribution Tails with Self-structuring Importance Samplers

Sampling from the Mean-Field Stationary Distribution

Probabilistic divide-and-conquer: a new exact simulation method, with integer partitions as an example

An Improved Algorithm for State Estimator Based on Maximum Normal Measurement Rate