Abstract:This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete probability distribution $(p_1, \dots, p_n)$, where the probabilities of the output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ of the sampling algorithm must be specified using at most $k$ bits of precision? We present a theoretical framework for formulating this problem and provide new techniques for finding sampling algorithms that are optimal both statistically (in the sense of sampling accuracy) and information-theoretically (in the sense of entropy consumption). We leverage these results to build a system that, for a broad family of measures of statistical accuracy, delivers a sampling algorithm whose expected entropy usage is minimal among those that induce the same distribution (i.e., is "entropy-optimal") and whose output distribution $(\hat{p}_1, \dots, \hat{p}_n)$ is a closest approximation to the target distribution $(p_1, \dots, p_n)$ among all entropy-optimal sampling algorithms that operate within the specified $k$-bit precision. This optimal approximate sampler is also a closer approximation than any (possibly entropy-suboptimal) sampler that consumes a bounded amount of entropy with the specified precision, a class which includes floating-point implementations of inversion sampling and related methods found in many software libraries. We evaluate the accuracy, entropy consumption, precision requirements, and wall-clock runtime of our optimal approximate sampling algorithms on a broad set of distributions, demonstrating the ways that they are superior to existing approximate samplers and establishing that they often consume significantly fewer resources than are needed by exact samplers.

Estimating a scalar log-concave random variable, using a silence set based probabilistic sampling

Estimating means of bounded random variables by betting

Adaptive complexity of log-concave sampling

Noisy recovery from random linear observations: Sharp minimax rates under elliptical constraints

Optimal Approximate Sampling from Discrete Probability Distributions

Instance-Optimality in I/O-Efficient Sampling and Sequential Estimation

Maximum likelihood estimation of the parameters of the inverse Gaussian distribution using maximum rank set sampling with unequal samples

An Explicit Sampling Dependent Spectral Error Bound for Column Subset Selection

Query lower bounds for log-concave sampling

Faster Diffusion Sampling with Randomized Midpoints: Sequential and Parallel

Log-Concave Sampling on Compact Supports: A Versatile Proximal Framework

Subsampled Optimization: Statistical Guarantees, Mean Squared Error Approximation, and Sampling Method

Sampling from the Mean-Field Stationary Distribution

Nonasymptotic Bounds for Suboptimal Importance Sampling

Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity - the Strongly Convex Case

A Stochastic Sampling Approach to Privacy

Sampling Approximately Low-Rank Ising Models: MCMC meets Variational Methods

Global Optimization Via Schrödinger-Föllmer Diffusion.

A Statistical Perspective of Sampling Scores for Linear Regression.

On the near-optimality of betting confidence sets for bounded means

Random points are good for universal discretization