Abstract:We describe Venture, an interactive virtual machine for probabilistic programming that aims to be sufficiently expressive, extensible, and efficient for general-purpose use. Like Church, probabilistic models and inference problems in Venture are specified via a Turing-complete, higher-order probabilistic language descended from Lisp. Unlike Church, Venture also provides a compositional language for custom inference strategies built out of scalable exact and approximate techniques. We also describe four key aspects of Venture's implementation that build on ideas from probabilistic graphical models. First, we describe the stochastic procedure interface (SPI) that specifies and encapsulates primitive random variables. The SPI supports custom control flow, higher-order probabilistic procedures, partially exchangeable sequences and ``likelihood-free'' stochastic simulators. It also supports external models that do inference over latent variables hidden from Venture. Second, we describe probabilistic execution traces (PETs), which represent execution histories of Venture programs. PETs capture conditional dependencies, existential dependencies and exchangeable coupling. Third, we describe partitions of execution histories called scaffolds that factor global inference problems into coherent sub-problems. Finally, we describe a family of stochastic regeneration algorithms for efficiently modifying PET fragments contained within scaffolds. Stochastic regeneration linear runtime scaling in cases where many previous approaches scaled quadratically. We show how to use stochastic regeneration and the SPI to implement general-purpose inference strategies such as Metropolis-Hastings, Gibbs sampling, and blocked proposals based on particle Markov chain Monte Carlo and mean-field variational inference techniques.

Accelerating Inference: towards a full Language, Compiler and Hardware stack

Simplifying debiased inference via automatic differentiation and probabilistic programming

Scaling Exact Inference for Discrete Probabilistic Programs

An Open-Source ML-Based Full-Stack Optimization Framework for Machine Learning Accelerators

Inference Compilation and Universal Probabilistic Programming

Compiling Halide Programs to Push-Memory Accelerators

Next-generation Probabilistic Computing Hardware with 3D MOSAICs, Illusion Scale-up, and Co-design

Venture: a higher-order probabilistic programming platform with programmable inference

LLM-Aided Compilation for Tensor Accelerators

SpecPIM: Accelerating Speculative Inference on PIM-Enabled System Via Architecture-Dataflow Co-Exploration

A Scala Based Framework for Developing Acceleration Systems with FPGAs

Structured Factored Inference: A Framework for Automated Reasoning in Probabilistic Programming Languages

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

PASS: An Asynchronous Probabilistic Processor for Next Generation Intelligence

Bring Your Own Codegen to Deep Learning Compiler

A Fast Convoluted Story: Scaling Probabilistic Inference for Integer Arithmetic

Restoring the Broken Covenant Between Compilers and Deep Learning Accelerators

Swift: Compiled Inference for Probabilistic Programming Languages.

Relay: A High-Level Compiler for Deep Learning

HIR: An MLIR-based Intermediate Representation for Hardware Accelerator Description