Abstract:Variational inference with normalizing flows (NFs) is an increasingly popular alternative to MCMC methods. In particular, NFs based on coupling layers (Real NVPs) are frequently used due to their good empirical performance. In theory, increasing the depth of normalizing flows should lead to more accurate posterior approximations. However, in practice, training deep normalizing flows for approximating high-dimensional posterior distributions is often infeasible due to the high variance of the stochastic gradients. In this work, we show that previous methods for stabilizing the variance of stochastic gradient descent can be insufficient to achieve stable training of Real NVPs. As the source of the problem, we identify that, during training, samples often exhibit unusual high values. As a remedy, we propose a combination of two methods: (1) soft-thresholding of the scale in Real NVPs, and (2) a bijective soft log transformation of the samples. We evaluate these and other previously proposed modification on several challenging target distributions, including a high-dimensional horseshoe logistic regression model. Our experiments show that with our modifications, stable training of Real NVPs for posteriors with several thousand dimensions is possible, allowing for more accurate marginal likelihood estimation via importance sampling. Moreover, we evaluate several common training techniques and architecture choices and provide practical advise for training NFs for high-dimensional variational inference.

Normalizing Flows for High-Dimensional Detector Simulations

Unifying Simulation and Inference with Normalizing Flows

Inductive Simulation of Calorimeter Showers with Normalizing Flows

CaloFlow: Fast and Accurate Generation of Calorimeter Showers with Normalizing Flows

Convolutional L2LFlows: Generating Accurate Showers in Highly Granular Calorimeters Using Convolutional Normalizing Flows

L2LFlows: Generating High-Fidelity 3D Calorimeter Images

Normalizing Flows for Domain Adaptation when Identifying $Λ$ Hyperon Events

Improving $Λ$ Signal Extraction with Domain Adaptation via Normalizing Flows

Free-form Flows: Make Any Architecture a Normalizing Flow

Generative machine learning for detector response modeling with a conditional normalizing flow

Anomaly detection with flow-based fast calorimeter simulators

Tensorizing flows: a tool for variational inference

CaloDREAM -- Detector Response Emulation via Attentive flow Matching

Stable Training of Normalizing Flows for High-dimensional Variational Inference

One flow to correct them all: improving simulations in high-energy physics with a single normalising flow and a switch

Chained Quantile Morphing with Normalizing Flows

Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed

Piecewise Normalizing Flows