Abstract:Though machine learning algorithms are able to achieve pattern recognition from the correlation between data and labels, the presence of spurious features in the data decreases the robustness of these learned relationships with respect to varied testing environments. This is known as out-of-distribution (OoD) generalization problem. Recently, invariant risk minimization (IRM) attempts to tackle this issue by penalizing predictions based on the unstable spurious features in the data collected from different environments. However, similar to domain adaptation or domain generalization, a prevalent non-trivial limitation in these works is that the environment information is assigned by human specialists, i.e. a priori, or determined heuristically. However, an inappropriate group partitioning can dramatically deteriorate the OoD generalization and this process is expensive and time-consuming. To deal with this issue, we propose a novel theoretically principled min-max framework to iteratively construct a worst-case splitting, i.e. creating the most challenging environment splittings for the backbone learning paradigm (e.g. IRM) to learn the robust feature representation. We also design a differentiable training strategy to facilitate the feasible gradient- based computation. Numerical experiments show that our algorithmic framework has achieved superior and stable performance in various datasets, such as Colored MNIST and Punctuated Stanford sentiment treebank (SST). Furthermore, we also find our algorithm to be robust even to a strong data poisoning attack. To the best of our knowledge, this is one of the first to adopt differentiable environment splitting method to enable stable predictions across environments without environment index information, which achieves the state-of-the-art performance on datasets with strong spurious correlation, such as Colored MNIST.

Sequential Invariant Information Bottleneck

Counterfactual Supervision-Based Information Bottleneck for Out-of-Distribution Generalization

Gated information bottleneck for generalization in sequential environments

Invariant Information Bottleneck for Domain Generalization.

Recognizable Information Bottleneck

Provably Invariant Learning Without Domain Information.

Invariant Graph Learning Meets Information Bottleneck for Out-of-Distribution Generalization

Elastic Information Bottleneck

Sequential Information Bottleneck Network For Rul Prediction

TIB: Detecting Unknown Objects via Two-Stream Information Bottleneck

Generalized Information Bottleneck for Gaussian Variables

Individual and Structural Graph Information Bottlenecks for Out-of-Distribution Generalization

Adversarial Invariant Learning.

Caveats for information bottleneck in deterministic scenarios

PAC-Bayes Information Bottleneck

Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix

Distributed Variational Information Bottleneck for IOT Environments

Tackling Distribution Shifts in Task-Oriented Communication with Information Bottleneck

Opportunistic Information-Bottleneck for Goal-oriented Feature Extraction and Communication

Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck

Towards a Theoretical Framework of Out-of-Distribution Generalization