Abstract:Probabilistic Circuits (PCs) are a promising avenue for probabilistic modeling. They combine advantages of probabilistic graphical models (PGMs) with those of neural networks (NNs). Crucially, however, they are tractable probabilistic models, supporting efficient and exact computation of many probabilistic inference queries, such as marginals and MAP. Further, since PCs are structured computation graphs, they can take advantage of deep-learning-style parameter updates, which greatly improves their scalability. However, this innovation also makes PCs prone to overfitting, which has been observed in many standard benchmarks. Despite the existence of abundant regularization techniques for both PGMs and NNs, they are not effective enough when applied to PCs. Instead, we re-think regularization for PCs and propose two intuitive techniques, data softening and entropy regularization, that both take advantage of PCs' tractability and still have an efficient implementation as a computation graph. Specifically, data softening provides a principled way to add uncertainty in datasets in closed form, which implicitly regularizes PC parameters. To learn parameters from a softened dataset, PCs only need linear time by virtue of their tractability. In entropy regularization, the exact entropy of the distribution encoded by a PC can be regularized directly, which is again infeasible for most other density estimation models. We show that both methods consistently improve the generalization performance of a wide variety of PCs. Moreover, when paired with a simple PC structure, we achieved state-of-the-art results on 10 out of 20 standard discrete density estimation benchmarks.

Tractable Regularization of Probabilistic Circuits

Sparse Probabilistic Circuits via Pruning and Growing

Scaling Up Probabilistic Circuits by Latent Variable Distillation

Soft Learning Probabilistic Circuits

Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits

Restructuring Tractable Probabilistic Circuits

Probabilistic Neural Circuits

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Characteristic Circuits

Probabilistic Circuits with Constraints via Convex Optimization

Polynomial Semantics of Tractable Probabilistic Circuits

Testing Probabilistic Circuits

On the Expressive Power of Tree-Structured Probabilistic Circuits

Probabilistic Circuits for Cumulative Distribution Functions

Scaling Tractable Probabilistic Circuits: A Systems Perspective

Mixture Bayesian Regularization of PCR Model and Soft Sensing Application

Probabilistic Generating Circuits -- Demystified

Building Expressive and Tractable Probabilistic Generative Models: A Review

Sum of Squares Circuits

Optimal Transport for Probabilistic Circuits