Abstract:The fruits of science are relationships made comprehensible, often by way of approximation. While deep learning is an extremely powerful way to find relationships in data, its use in science has been hindered by the difficulty of understanding the learned relationships. The Information Bottleneck (IB) is an information theoretic framework for understanding a relationship between an input and an output in terms of a trade-off between the fidelity and complexity of approximations to the relationship. Here we show that a crucial modification -- distributing bottlenecks across multiple components of the input -- opens fundamentally new avenues for interpretable deep learning in science. The Distributed Information Bottleneck throttles the downstream complexity of interactions between the components of the input, deconstructing a relationship into meaningful approximations found through deep learning without requiring custom-made datasets or neural network architectures. Applied to a complex system, the approximations illuminate aspects of the system's nature by restricting -- and monitoring -- the information about different components incorporated into the approximation. We demonstrate the Distributed IB's explanatory utility in systems drawn from applied mathematics and condensed matter physics. In the former, we deconstruct a Boolean circuit into approximations that isolate the most informative subsets of input components without requiring exhaustive search. In the latter, we localize information about future plastic rearrangement in the static structure of a sheared glass, and find the information to be more or less diffuse depending on the system's preparation. By way of a principled scheme of approximations, the Distributed IB brings much-needed interpretability to deep learning and enables unprecedented analysis of information flow through a system.

Distributed Variational Information Bottleneck for IOT Environments

Blockchain-Enabled Variational Information Bottleneck for IoT Networks

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

A Distributed Privacy-Preserving Framework for Deep Learning with Edge-Cloud Computing.

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Blockchain-Enabled Variational Information Bottleneck for Data Extraction Based on Mutual Information in Internet of Vehicles

A Variational Information Bottleneck Method for Network Intrusion Detection

Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training

Visualizing Information Bottleneck through Variational Inference

Disentangled Representation Learning with Transmitted Information Bottleneck

Elastic Information Bottleneck

The Distributed Information Bottleneck reveals the explanatory structure of complex systems

Opportunistic Information-Bottleneck for Goal-oriented Feature Extraction and Communication

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression

Differentiable Information Bottleneck for Deterministic Multi-view Clustering

Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Deep Learning and the Information Bottleneck Principle

Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck

Information Bottleneck Methods for Distributed Learning

Disentangled Variational Information Bottleneck for Multiview Representation Learning