Abstract:One of the most studied problems in machine learning is finding reasonable constraints that guarantee the generalization of a learning algorithm. These constraints are usually expressed as some simplicity assumptions on the target. For instance, in the Vapnik-Chervonenkis (VC) theory the space of possible hypotheses is considered to have a limited VC dimension. In this paper, the constraint on the entropy $H(X)$ of the input variable $X$ is studied as a simplicity assumption. It is proven that the sample complexity to achieve an $\epsilon$-$\delta$ Probably Approximately Correct (PAC) hypothesis is bounded by $\frac{2^{ \left.6H(X)\middle/\epsilon\right.}+\log{\frac{1}{\delta}}}{\epsilon^2}$ which is sharp up to the $\frac{1}{\epsilon^2}$ factor. Morever, it is shown that if a feature learning process is employed to learn the compressed representation from the dataset, this bound no longer exists. These findings have important implications on the Information Bottleneck (IB) theory which had been utilized to explain the generalization power of Deep Neural Networks (DNNs), but its applicability for this purpose is currently under debate by researchers. In particular, this is a rigorous proof for the previous heuristic that compressed representations are exponentially easier to be learned. However, our analysis pinpoints two factors preventing the IB, in its current form, to be applicable in studying neural networks. Firstly, the exponential dependence of sample complexity on $\frac{1}{\epsilon}$, which can lead to a dramatic effect on the bounds in practical applications when $\epsilon$ is small. Secondly, our analysis reveals that arguments based on input compression are inherently insufficient to explain generalization of methods like DNNs in which the features are also learned using available data.

On the Perceptron's Compression

Significant improvement of lossy compression rate and speed of HPC data using perceptron parallelized compression

The Perceptron Algorithm: Image and Signal Decomposition, Compression, and Analysis by Iterative Gaussian Blurring

On the Impact of Perceptual Compression on Deep Learning

Model compression as constrained optimization, with application to neural nets. Part V: combining compressions

On Accelerated Perceptrons and Beyond

Compression, Generalization and Learning

The Helmholtz Method: Using Perceptual Compression to Reduce Machine Learning Complexity

Learned Image Compression for Machine Perception

Perceptron Mistake Bounds

Hyper-Compression: Model Compression via Hyperfunction

A simple connection from loss flatness to compressed representations in neural networks

Machine Perceptual Quality: Evaluating the Impact of Severe Lossy Compression on Audio and Image Models

Exploring Compressed Image Representation as a Perceptual Proxy: A Study

Do Compressed Representations Generalize Better?

MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression

Deep Perceptual Compression

Model compression as constrained optimization, with application to neural nets. Part II: quantization

What Do Compressed Deep Neural Networks Forget?

Compressive Visual Representations

Towards Explaining Deep Neural Network Compression Through a Probabilistic Latent Space