Abstract:The Information Bottleneck (IB) method is an information theoretical framework to design a parsimonious and tunable feature-extraction mechanism, such that the extracted features are maximally relevant to a specific learning or inference task. Despite its theoretical value, the IB is based on a functional optimization problem that admits a closed form solution only on specific cases (e.g., Gaussian distributions), making it difficult to be applied in most applications, where it is necessary to resort to complex and approximated variational implementations. To overcome this limitation, we propose an approach to adapt the closed-form solution of the Gaussian IB to a general task. Whichever is the inference task to be performed by a (possibly deep) neural-network, the key idea is to opportunistically design a regression sub-task, embedded in the original problem, where we can safely assume a (joint) multivariate normality between the sub-task's inputs and outputs. In this way we can exploit a fixed and pre-trained neural network to process the input data, using a tunable number of features, to trade data-size and complexity for accuracy. This approach is particularly useful every time a device needs to transmit data (or features) to a server that has to fulfil an inference task, as it provides a principled way to extract the most relevant features for the task to be executed, while looking for the best trade-off between the size of the feature vector to be transmitted, inference accuracy, and complexity. Extensive simulation results testify the effectiveness of the proposed method and encourage to further investigate this research line.

Structured IB: Improving Information Bottleneck with Structured Feature Learning

Structured Label Inference for Visual Understanding.

Opportunistic Information-Bottleneck for Goal-oriented Feature Extraction and Communication

Information-Ordered Bottlenecks for Adaptive Semantic Compression

IB-AdCSCNet:Adaptive Convolutional Sparse Coding Network Driven by Information Bottleneck

Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck

An Information Bottleneck Approach for Multi-View Feature Selection

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

$S^{2}$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning.

Elastic Information Bottleneck

Aggregated Learning: A Deep Learning Framework Based on Information-Bottleneck Vector Quantization

Statistically Valid Information Bottleneck via Multiple Hypothesis Testing

Information Bottleneck: Theory and Applications in Deep Learning

Towards a More Stable and General Subgraph Information Bottleneck

Justices for Information Bottleneck Theory

Graph Structure Learning with Variational Information Bottleneck

Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis

Gated information bottleneck for generalization in sequential environments

Cauchy-Schwarz Divergence Information Bottleneck for Regression

A Survey on Information Bottleneck

Information Bottleneck and Aggregated Learning