Fabian Zaiser,Andrzej S. Murawski,Luke Ong
Abstract:We present an exact Bayesian inference method for discrete statistical models, which can find exact solutions to a large class of discrete inference problems, even with infinite support and continuous priors. To express such models, we introduce a probabilistic programming language that supports discrete and continuous sampling, discrete observations, affine functions, (stochastic) branching, and conditioning on discrete events. Our key tool is probability generating functions: they provide a compact closed-form representation of distributions that are definable by programs, thus enabling the exact computation of posterior probabilities, expectation, variance, and higher moments. Our inference method is provably correct and fully automated in a tool called Genfer, which uses automatic differentiation (specifically, Taylor polynomials), but does not require computer algebra. Our experiments show that Genfer is often faster than the existing exact inference tools PSI, Dice, and Prodigy. On a range of real-world inference problems that none of these exact tools can solve, Genfer's performance is competitive with approximate Monte Carlo methods, while avoiding approximation errors.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to achieve exact Bayesian inference in discrete statistical models. Specifically, the author proposes a method that can find exact solutions for a large class of discrete inference problems with infinite support and continuous priors. This method is particularly suitable for time - series models of count data, such as autoregressive models, hidden Markov models, and change - point models.
### Core problems of the paper
1. **Challenges of exact inference**: In Bayesian statistics, calculating the posterior distribution is usually difficult because analytical solutions often do not exist or are difficult to solve. Therefore, approximate methods such as Markov Chain Monte Carlo (MCMC) or variational inference are usually required. However, these methods have approximation errors.
2. **Exact inference for specific models**: The author identifies a class of discrete models, especially time - series models of count data, which can achieve exact inference through probability generating functions (PGFs). This provides exact solutions for some complex models, not just simple ones.
### Solutions
1. **Probability generating functions (PGFs)**: The author uses probability generating functions to represent distributions. These functions can provide a compact closed - form representation, thereby enabling the exact calculation of posterior probabilities, expectations, variances, and higher - order moments.
2. **Probability programming languages**: The author introduces a probability programming language (SGCL) that supports discrete and continuous sampling, discrete observations, affine functions, (random) branching, and conditioning based on discrete events. The design of this language enables programs to be automatically translated into generating functions for further exact inference.
3. **Automation tools**: The author has developed a tool named Genfer. This tool uses automatic differentiation (especially Taylor polynomials) to calculate generating functions and their derivatives, thus avoiding the complexity of computer algebra. Experiments show that Genfer is faster than existing exact inference tools in many benchmarks and has performance comparable to approximate Monte Carlo methods when dealing with practical problems, but without approximation errors.
### Main contributions
1. **Wide applicability**: This method is applicable to discrete models with infinite support, especially time - series models of count data, such as autoregressive models, Bayesian change - point models, mixture models, and hidden Markov models.
2. **Flexibility**: Through the probability programming language, users can flexibly define models, supporting random branching, continuous and discrete priors, discrete observations, and conditioning based on discrete variables.
3. **Automatic translation**: Every program written in this language can be automatically translated into a generating function, thereby representing the posterior distribution in a provably correct manner.
4. **Optimization tool**: The Genfer tool performs well in many benchmarks, especially when dealing with practical problems, its performance is comparable to approximate Monte Carlo methods, but without approximation errors.
### Related work
- **Limitations of existing tools**: Existing exact inference tools such as Dice, SPPL, and PSI either only support limited - support distributions or cannot guarantee successful inference.
- **Applications of generating functions**: Generating functions are used in probability theory to study random variables with infinite support, for example, in branching processes. In Bayesian inference, probability generating function circuits (PGCs) use generating functions to improve the expressive power of probability circuits, but cannot handle random variables with infinite support.
### Limitations
- **Computational complexity**: Even for probability programs that only involve finite discrete distributions, exact posterior inference is PSPACE - hard. Therefore, this method needs to limit the class of probability programs with support, for example, prohibiting nonlinear transformations and observations of continuous variables.
- **Performance**: The running time of inference is polynomial in the number of values observed in the program and exponential in the number of program variables. Nevertheless, this method still performs well in many practical models and can achieve efficient exact inference in practice.
In conclusion, this paper provides an exact Bayesian inference method for a class of complex discrete models by introducing probability generating functions and probability programming languages, solving the approximation error problem existing in existing methods.