Abstract:One of the most studied problems in machine learning is finding reasonable constraints that guarantee the generalization of a learning algorithm. These constraints are usually expressed as some simplicity assumptions on the target. For instance, in the Vapnik-Chervonenkis (VC) theory the space of possible hypotheses is considered to have a limited VC dimension. In this paper, the constraint on the entropy $H(X)$ of the input variable $X$ is studied as a simplicity assumption. It is proven that the sample complexity to achieve an $\epsilon$-$\delta$ Probably Approximately Correct (PAC) hypothesis is bounded by $\frac{2^{ \left.6H(X)\middle/\epsilon\right.}+\log{\frac{1}{\delta}}}{\epsilon^2}$ which is sharp up to the $\frac{1}{\epsilon^2}$ factor. Morever, it is shown that if a feature learning process is employed to learn the compressed representation from the dataset, this bound no longer exists. These findings have important implications on the Information Bottleneck (IB) theory which had been utilized to explain the generalization power of Deep Neural Networks (DNNs), but its applicability for this purpose is currently under debate by researchers. In particular, this is a rigorous proof for the previous heuristic that compressed representations are exponentially easier to be learned. However, our analysis pinpoints two factors preventing the IB, in its current form, to be applicable in studying neural networks. Firstly, the exponential dependence of sample complexity on $\frac{1}{\epsilon}$, which can lead to a dramatic effect on the bounds in practical applications when $\epsilon$ is small. Secondly, our analysis reveals that arguments based on input compression are inherently insufficient to explain generalization of methods like DNNs in which the features are also learned using available data.

Compressed Learning With Regular Concept

Structure discovery in PAC-learning by random projections

Compressed representation of Learning Spaces

A PAC learning approach to one-bit compressed sensing

Few-Shot Non-Parametric Learning with Deep Latent Variable Model

Compression, Generalization and Learning

A Unified Active and Semi-Supervised Learning Framework for Image Compression

On statistical learning via the lens of compression

Non-Convex SGD Learns Halfspaces with Adversarial Label Noise

Hyperspherical Consistency Regularization.

Enhancing PAC Learning of Half spaces Through Robust Optimization Techniques

Smoothed Analysis for Learning Concepts with Low Intrinsic Dimension

Efficient PAC Learning of Halfspaces with Constant Malicious Noise Rate

On PAC Learning Halfspaces in Non-interactive Local Privacy Model with Public Unlabeled Data

On the complexity of PAC learning in Hilbert spaces

Consistency-Contrast Learning for Conceptual Coding

List Sample Compression and Uniform Convergence

Compressive Visual Representations

Do Compressed Representations Generalize Better?

Adversarially Robust PAC Learnability of Real-Valued Functions

Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Bounds