Abstract:Given data drawn from an unknown distribution, $D$, to what extent is it possible to ``amplify'' this dataset and output an even larger set of samples that appear to have been drawn from $D$? We formalize this question as follows: an $(n,m)$ $\text{amplification procedure}$ takes as input $n$ independent draws from an unknown distribution $D$, and outputs a set of $m > n$ ``samples''. An amplification procedure is valid if no algorithm can distinguish the set of $m$ samples produced by the amplifier from a set of $m$ independent draws from $D$, with probability greater than $2/3$. Perhaps surprisingly, in many settings, a valid amplification procedure exists, even when the size of the input dataset, $n$, is significantly less than what would be necessary to learn $D$ to non-trivial accuracy. Specifically we consider two fundamental settings: the case where $D$ is an arbitrary discrete distribution supported on $\le k$ elements, and the case where $D$ is a $d$-dimensional Gaussian with unknown mean, and fixed covariance. In the first case, we show that an $\left(n, n + \Theta(\frac{n}{\sqrt{k}})\right)$ amplifier exists. In particular, given $n=O(\sqrt{k})$ samples from $D$, one can output a set of $m=n+1$ datapoints, whose total variation distance from the distribution of $m$ i.i.d. draws from $D$ is a small constant, despite the fact that one would need quadratically more data, $n=\Theta(k)$, to learn $D$ up to small constant total variation distance. In the Gaussian case, we show that an $\left(n,n+\Theta(\frac{n}{\sqrt{d}} )\right)$ amplifier exists, even though learning the distribution to small constant total variation distance requires $\Theta(d)$ samples. In both the discrete and Gaussian settings, we show that these results are tight, to constant factors. Beyond these results, we formalize a number of curious directions for future research along this vein.

The Sample Complexity of Smooth Boosting and the Tightness of the Hardcore Theorem

The Cost of Parallelizing Boosting

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-$\ell_1$-Norm Interpolated Classifiers

Optimal Parallelization of Boosting

Distribution-Specific Agnostic Boosting

Boosting, Voting Classifiers and Randomized Sample Compression Schemes

On the Statistical Complexity of Sample Amplification

Sampling numbers of smoothness classes via $\ell^1$-minimization

Analysis of boosting algorithms using the smooth margin function

The Many Faces of Optimal Weak-to-Strong Learning

On the complexity of sampling from shallow Brownian circuits

Multiclass Boosting: Simple and Intuitive Weak Learning Criteria

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

Distribution-Dependent Sample Complexity of Large Margin Learning

Pseudorandomness, symmetry, smoothing: I

Sample Amplification: Increasing Dataset Size even when Learning is Impossible

How many samples are needed to leverage smoothness?

A Strong Composition Theorem for Junta Complexity and the Boosting of Property Testers

Unconditional Quantum Advantage for Sampling with Shallow Circuits

Sharper Bounds for Chebyshev Moment Matching with Applications to Differential Privacy and Beyond

Sample-Efficient Agnostic Boosting