Abstract:We present novel reductions from sample compression schemes in multiclass classification, regression, and adversarially robust learning settings to binary sample compression schemes. Assuming we have a compression scheme for binary classes of size $f(d_\mathrm{VC})$, where $d_\mathrm{VC}$ is the VC dimension, then we have the following results: (1) If the binary compression scheme is a majority-vote or a stable compression scheme, then there exists a multiclass compression scheme of size $O(f(d_\mathrm{G}))$, where $d_\mathrm{G}$ is the graph dimension. Moreover, for general binary compression schemes, we obtain a compression of size $O(f(d_\mathrm{G})\log|Y|)$, where $Y$ is the label space. (2) If the binary compression scheme is a majority-vote or a stable compression scheme, then there exists an $\epsilon$-approximate compression scheme for regression over $[0,1]$-valued functions of size $O(f(d_\mathrm{P}))$, where $d_\mathrm{P}$ is the pseudo-dimension. For general binary compression schemes, we obtain a compression of size $O(f(d_\mathrm{P})\log(1/\epsilon))$. These results would have significant implications if the sample compression conjecture, which posits that any binary concept class with a finite VC dimension admits a binary compression scheme of size $O(d_\mathrm{VC})$, is resolved (Littlestone and Warmuth, 1986; Floyd and Warmuth, 1995; Warmuth, 2003). Our results would then extend the proof of the conjecture immediately to other settings. We establish similar results for adversarially robust learning and also provide an example of a concept class that is robustly learnable but has no bounded-size compression scheme, demonstrating that learnability is not equivalent to having a compression scheme independent of the sample size, unlike in binary classification, where compression of size $2^{O(d_\mathrm{VC})}$ is attainable (Moran and Yehudayoff, 2016).

Bounding Embeddings of VC Classes into Maximum Classes

Unlabelled Sample Compression Schemes for Intersection-Closed Classes and Extremal Classes

Sample Compression Scheme Reductions

A Labelled Sample Compression Scheme of Size at Most Quadratic in the VC Dimension

Multiclass Learnability Does Not Imply Sample Compression

A New Lower Bound for Agnostic Learning with Sample Compression Schemes

Unlabeled sample compression schemes for oriented matroids

CS 6781-Theoretical Foundations of Machine Learning Lecture 5 : PAC Sample Complexity Proof

Computing the Vapnik Chervonenkis Dimension for Non-Discrete Settings

Error Bounds for Real Function Classes Based on Discretized Vapnik-Chervonenkis Dimensions.

The Sample Complexity of Multi-Distribution Learning for VC Classes

Sample Complexity Result for Multi-category Classifiers of Bounded Variation

VC-Dimension and Distance Chains in $\mathbb{F}_q^d$

A PAC learning approach to one-bit compressed sensing

Measurability Aspects of the Compactness Theorem for Sample Compression Schemes

Lower and Upper Bounds on the VC-Dimension of Tensor Network Models

On statistical learning via the lens of compression

New Bounds on Quantum Sample Complexity of Measurement Classes

Prediction, Learning, Uniform Convergence, and Scale-sensitive Dimensions

Vapnik-Chervonenkis Dimension and Density on Johnson and Hamming Graphs

On a Combinatorial Problem Arising in Machine Teaching