Abstract:We introduce marginalization models (MAMs), a new family of generative models for high-dimensional discrete data. They offer scalable and flexible generative modeling by explicitly modeling all induced marginal distributions. Marginalization models enable fast approximation of arbitrary marginal probabilities with a single forward pass of the neural network, which overcomes a major limitation of arbitrary marginal inference models, such as any-order autoregressive models. MAMs also address the scalability bottleneck encountered in training any-order generative models for high-dimensional problems under the context of energy-based training, where the goal is to match the learned distribution to a given desired probability (specified by an unnormalized log-probability function such as energy or reward function). We propose scalable methods for learning the marginals, grounded in the concept of "marginalization self-consistency". We demonstrate the effectiveness of the proposed model on a variety of discrete data distributions, including images, text, physical systems, and molecules, for maximum likelihood and energy-based training settings. MAMs achieve orders of magnitude speedup in evaluating the marginal probabilities on both settings. For energy-based training tasks, MAMs enable any-order generative modeling of high-dimensional problems beyond the scale of previous methods. Code is available at <a class="link-external link-https" href="https://github.com/PrincetonLIPS/MaM" rel="external noopener nofollow">this https URL</a>.

On the Efficient Marginalization of Probabilistic Sequence Models

Predictive Querying for Autoregressive Neural Sequence Models

Probabilistic Modeling for Sequences of Sets in Continuous-Time

Exchangeable Sequence Models Quantify Uncertainty Over Latent Concepts

Mixing Times and Structural Inference for Bernoulli Autoregressive Processes

User-Dependent Neural Sequence Models for Continuous-Time Event Data

History Marginalization Improves Forecasting in Variational Recurrent Neural Networks

Generative Marginalization Models

Probabilistic Time Series Modeling with Decomposable Denoising Diffusion Model

Scalable marginalization of correlated latent variables with applications to learning particle interaction kernels

Multimarginal generative modeling with stochastic interpolants

Deep Learning for Marginal Bayesian Posterior Inference with Recurrent Neural Networks

Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity

Adversarial Resilience in Sequential Prediction via Abstention

Probability Tools for Sequential Random Projection

A Modularized Efficient Framework for Non-Markov Time Series Estimation

Efficient Learning Via Simulation: A Marginalized Resample-Move Approach

Sequential Inference of Hospitalization Electronic Health Records Using Probabilistic Models

Efficiently Answering Durability Prediction Queries

Efficient stochastic optimisation by unadjusted Langevin Monte Carlo. Application to maximum marginal likelihood and empirical Bayesian estimation

Robust parametric inference for finite Markov chains