Abstract:This survey is for the remembrance of one of the creators of the information bottleneck theory, Prof. Naftali Tishby, passing away at the age of 68 on August, 2021. Information bottleneck (IB), a novel information theoretic approach for pattern analysis and representation learning, has gained widespread popularity since its birth in 1999. It provides an elegant balance between data compression and information preservation, and improves its prediction or representation ability accordingly. This survey summarizes both the theoretical progress and practical applications on IB over the past 20-plus years, where its basic theory, optimization, extensive models and task-oriented algorithms are systematically explored. Existing IB methods are roughly divided into two parts: traditional and deep IB, where the former contains the IBs optimized by traditional machine learning analysis techniques without involving any neural networks, and the latter includes the IBs involving the interpretation, optimization and improvement of deep neural works (DNNs). Specifically, based on the technique taxonomy, traditional IBs are further classified into three categories: Basic, Informative and Propagating IB; While the deep IBs, based on the taxonomy of problem settings, contain Debate: Understanding DNNs with IB, Optimizing DNNs Using IB, and DNN-based IB methods. Furthermore, some potential issues deserving future research are discussed. This survey attempts to draw a more complete picture of IB, from which the subsequent studies can benefit.

The Cardinality Bound on the Information Bottleneck Representations is Tight

Generalized Information Bottleneck for Gaussian Variables

The Double-Sided Information Bottleneck Function

Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Recognizable Information Bottleneck

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

Efficient and Provably Convergent Computation of Information Bottleneck: A Semi-Relaxed Approach

Elastic Information Bottleneck

Caveats for information bottleneck in deterministic scenarios

The deterministic information bottleneck

The Mutual Information In The Vicinity of Capacity-Achieving Input Distributions

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

PAC-Bayes Information Bottleneck

Lower Bounds on Mutual Information for Linear Codes Transmitted over Binary Input Channels, and for Information Combining

Beyond Countable Alphabets: an Extension of the Information-Spectrum Approach.

Data Efficiency, Dimensionality Reduction, and the Generalized Symmetric Information Bottleneck

An Achievable and Analytic Solution to Information Bottleneck for Gaussian Mixtures

Statistically Valid Information Bottleneck via Multiple Hypothesis Testing

A Survey on Information Bottleneck

Justices for Information Bottleneck Theory