Abstract:We propose a novel framework for uncertainty quantification via information bottleneck (IB-UQ) for scientific machine learning tasks, including deep neural network (DNN) regression and neural operator learning (DeepONet). Specifically, we incorporate the bottleneck by a confidence-aware encoder, which encodes inputs into latent representations according to the confidence of the input data belonging to the region where training data is located, and utilize a Gaussian decoder to predict means and variances of outputs conditional on representation variables. Furthermore, we propose a data augmentation based information bottleneck objective which can enhance the quantification quality of the extrapolation uncertainty, and the encoder and decoder can be both trained by minimizing a tractable variational bound of the objective. In comparison to uncertainty quantification (UQ) methods for scientific learning tasks that rely on Bayesian neural networks with Hamiltonian Monte Carlo posterior estimators, the model we propose is computationally efficient, particularly when dealing with large-scale datasets. The effectiveness of the IB-UQ model has been demonstrated through several representative examples, such as regression for discontinuous functions, learning nonlinear operators for partial differential equations, and a large-scale climate map model. The experimental results indicate that the IB-UQ model can handle noisy data, generate robust predictions, and provide confident uncertainty evaluation for out-of-distribution data.

Exploring the Trade-Off in the Variational Information Bottleneck for Regression with a Single Training Run

Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training

Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss

Uncertainty in the Variational Information Bottleneck

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

Cauchy-Schwarz Divergence Information Bottleneck for Regression

Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Visualizing Information Bottleneck through Variational Inference

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Learning Robust Variational Information Bottleneck with Reference

Generalized Information Bottleneck for Gaussian Variables

Statistically Valid Information Bottleneck via Multiple Hypothesis Testing

Cell Variational Information Bottleneck Network

Elastic Information Bottleneck

Caveats for information bottleneck in deterministic scenarios

IB-UQ: Information bottleneck based uncertainty quantification for neural function regression and neural operator learning

Multi-Task Variational Information Bottleneck

Generalized Variational Inference Via Optimal Transport

Relevant sparse codes with variational information bottleneck

Efficient and Provably Convergent Computation of Information Bottleneck: A Semi-Relaxed Approach