Implicit Normalizing Flows

Cheng Lu,Jianfei Chen,Chongxuan Li,Qiuhao Wang,Jun Zhu

DOI: https://doi.org/10.48550/arxiv.2103.09527

2021-01-01

Abstract:Normalizing flows define a probability distribution by an explicit invertible transformation z=f(x). In this work, we present implicit normalizing flows (ImpFlows), which generalize normalizing flows by allowing the mapping to be implicitly defined by the roots of an equation F(z,x)=0. ImpFlows build on residual flows (ResFlows) with a proper balance between expressiveness and tractability. Through theoretical analysis, we show that the function space of ImpFlow is strictly richer than that of ResFlows. Furthermore, for any ResFlow with a fixed number of blocks, there exists some function that ResFlow has a non-negligible approximation error. However, the function is exactly representable by a single-block ImpFlow. We propose a scalable algorithm to train and draw samples from ImpFlows. Empirically, we evaluate ImpFlow on several classification and density modeling tasks, and ImpFlow outperforms ResFlow with a comparable amount of parameters on all the benchmarks.