Abstract:In recent several years, the information bottleneck (IB) principle provides an information-theoretic framework for deep multi-view clustering (MVC) by compressing multi-view observations while preserving the relevant information of multiple views. Although existing IB-based deep MVC methods have achieved huge success, they rely on variational approximation and distribution assumption to estimate the lower bound of mutual information, which is a notoriously hard and impractical problem in high-dimensional multi-view spaces. In this work, we propose a new differentiable information bottleneck (DIB) method, which provides a deterministic and analytical MVC solution by fitting the mutual information without the necessity of variational approximation. Specifically, we first propose to directly fit the mutual information of high-dimensional spaces by leveraging normalized kernel Gram matrix, which does not require any auxiliary neural estimator to estimate the lower bound of mutual information. Then, based on the new mutual information measurement, a deterministic multi-view neural network with analytical gradients is explicitly trained to parameterize IB principle, which derives a deterministic compression of input variables from different views. Finally, a triplet consistency discovery mechanism is devised, which is capable of mining the feature consistency, cluster consistency and joint consistency based on the deterministic and compact representations. Extensive experimental results show the superiority of our DIB method on 6 benchmarks compared with 13 state-of-the-art baselines.

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Disentangled Variational Information Bottleneck for Multiview Representation Learning

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression

A Variational Information Bottleneck Method for Network Intrusion Detection

Variational Distillation for Multi-View Learning

Visualizing Information Bottleneck through Variational Inference

Distributed Variational Information Bottleneck for IOT Environments

Class-Conditional Compression and Disentanglement: Bridging the Gap between Neural Networks and Naive Bayes Classifiers

Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss

Differentiable Information Bottleneck for Deterministic Multi-view Clustering

Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck

On Variational Bounds of Mutual Information

Variational Information Bottleneck on Vector Quantized Autoencoders

Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

Multi-Task Variational Information Bottleneck

Understanding Deep Learning by Revisiting Boltzmann Machines: an Information Geometry Approach

Uncertainty in the Variational Information Bottleneck

Deep Variational Metric Learning