Abstract:Compared to the wide array of advanced Monte Carlo methods supported by modern probabilistic programming languages (PPLs), PPL support for variational inference (VI) is less developed: users are typically limited to a predefined selection of variational objectives and gradient estimators, which are implemented monolithically (and without formal correctness arguments) in PPL backends. In this paper, we propose a more modular approach to supporting variational inference in PPLs, based on compositional program transformation. In our approach, variational objectives are expressed as programs, that may employ first-class constructs for computing densities of and expected values under user-defined models and variational families. We then transform these programs systematically into unbiased gradient estimators for optimizing the objectives they define. Our design enables modular reasoning about many interacting concerns, including automatic differentiation, density accumulation, tracing, and the application of unbiased gradient estimation strategies. Additionally, relative to existing support for VI in PPLs, our design increases expressiveness along three axes: (1) it supports an open-ended set of user-defined variational objectives, rather than a fixed menu of options; (2) it supports a combinatorial space of gradient estimation strategies, many not automated by today's PPLs; and (3) it supports a broader class of models and variational families, because it supports constructs for approximate marginalization and normalization (previously introduced only for Monte Carlo inference). We implement our approach in an extension to the Gen probabilistic programming system (<a class="link-external link-http" href="http://genjax.vi" rel="external noopener nofollow">this http URL</a>, implemented in JAX), and evaluate on several deep generative modeling tasks, showing minimal performance overhead vs. hand-coded implementations and performance competitive with well-established open-source PPLs.

Swift: Compiled Inference for Probabilistic Programming Languages.

Swift: Fast, Reliable, Loosely Coupled Parallel Computation

SPPL: Probabilistic Programming with Fast Exact Symbolic Inference

Probabilistic Programming with Programmable Variational Inference

flip-hoisting: Exploiting Repeated Parameters in Discrete Probabilistic Programs

Suspension Analysis and Selective Continuation-Passing Style for Universal Probabilistic Programming Languages

Inference Compilation and Universal Probabilistic Programming

Functional probabilistic programming for scalable Bayesian modelling

Compiling Stan to Generative Probabilistic Languages and Extension to Deep Probabilistic Programming

Semi-Symbolic Inference for Efficient Streaming Probabilistic Programming

SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

DynamicPPL: Stan-like Speed for Dynamic Probabilistic Models

An Introduction to Probabilistic Programming

BFiT: From Possible-World Semantics to Random-Evaluation Semantics in an Open Universe

Discrete-Continuous Mixtures in Probabilistic Programming: Generalized Semantics and Inference Algorithms.

Accelerating Inference: towards a full Language, Compiler and Hardware stack

Scaling Exact Inference for Discrete Probabilistic Programs

Composing inference algorithms as program transformations

Swift: A Register-Based Jit Compiler For Embedded Jvms

SpecPIM: Accelerating Speculative Inference on PIM-Enabled System Via Architecture-Dataflow Co-Exploration