Abstract:Suppose Alice has a distribution $P$ and Bob has a distribution $Q$. Alice wants to generate a sample $a\sim P$ and Bob a sample $b \sim Q$ such that $a = b$ with has as high of probability as possible. It is well-known that, by sampling from an optimal coupling between the distributions, Alice and Bob can achieve $Pr[a = b] = 1 - D_{TV}(P,Q)$, where $D_{TV}(P,Q)$ is the total variation distance. What if Alice and Bob must solve this same problem without communicating at all? Perhaps surprisingly, with access to public randomness, they can still achieve $Pr[a=b] \geq \frac{1-D_{TV}(P,Q)}{1+D_{TV}(P,Q)} \geq 1-2D_{TV}(P,Q)$. In fact, this bound can be obtained using a simple protocol based on the Weighted MinHash algorithm. In this work, we explore the communication-free coupling problem in greater depth. First, we show that an equally simple protocol based on Gumbel sampling matches the worst-case guarantees of the Weighted MinHash approach, but tends to perform better in practice. Conversely, we prove that both approaches are actually sharp: no communication-free protocol can achieve $Pr[a=b]>\frac{1-D_{TV}(P,Q)}{1+D_{TV}(P,Q)}$ in the worst-case. Finally, we prove that, for distributions over $n$ items, there exists a scheme that uses just $O(\log(n/\epsilon))$ bits of communication to achieve $Pr[a = b] = 1 - D_{TV}(P,Q) - \epsilon$, i.e. to essentially match optimal coupling. Beyond our theoretical results, we demonstrate an application of communication-free coupling to speculative decoding, a recent method for accelerating autoregressive large language models [Leviathan, Kalman, Matias, ICML 2023]. We show that communication-free protocols yield a variant of speculative decoding that we call Drafter-Invariant Speculative Decoding, which has the desirable property that the output of the method is fixed given a fixed random seed, regardless of what drafter is used for speculation.

Computing Low-Entropy Couplings for Large-Support Distributions

Minimum Entropy Coupling with Bottleneck

A Sparse MLE Approach for Joint Interference Mitigation and Data Recovery

A Tighter Approximation Guarantee for Greedy Minimum Entropy Coupling

Bounding Wasserstein distance with couplings

Scalable couplings for the random walk Metropolis algorithm

Quantized Minimum Error Entropy with Fiducial Points for Robust Regression.

Coupling without Communication and Drafter-Invariant Speculative Decoding

Insights into the Robustness of Minimum Error Entropy Estimation

Computing Marginal and Conditional Divergences between Decomposable Models with Applications

Distributed Matrix Computations with Low-weight Encodings

A Geometric Unification of Distributionally Robust Covariance Estimators: Shrinking the Spectrum by Inflating the Ambiguity Set

Byzantine-tolerant distributed learning of finite mixture models

Some Further Results on the Minimum Error Entropy Estimation.

DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification

OD-Stega: LLM-Based Near-Imperceptible Steganography via Optimized Distributions

Out-of-Distribution Detection using Maximum Entropy Coding

On the Low-Temperature MCMC threshold: the cases of sparse tensor PCA, sparse regression, and a geometric rule

Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity

Bipartite reweight-annealing algorithm to extract large-scale data of entanglement entropy and its derivative in high precision