Abstract:In recent years, we have witnessed a surge of Graph Neural Networks (GNNs), most of which can learn powerful representations in an end-to-end fashion with great success in many real-world applications. They have resemblance to Probabilistic Graphical Models (PGMs), but break free from some limitations of PGMs. By aiming to provide expressive methods for representation learning instead of computing marginals or most likely configurations, GNNs provide flexibility in the choice of information flowing rules while maintaining good performance. Despite their success and inspirations, they lack efficient ways to represent and learn higher-order relations among variables/nodes. More expressive higher-order GNNs which operate on k-tuples of nodes need increased computational resources in order to process higher-order tensors. We propose Factor Graph Neural Networks (FGNNs) to effectively capture higher-order relations for inference and learning. To do so, we first derive an efficient approximate Sum-Product loopy belief propagation inference algorithm for discrete higher-order PGMs. We then neuralize the novel message passing scheme into a Factor Graph Neural Network (FGNN) module by allowing richer representations of the message update rules; this facilitates both efficient inference and powerful end-to-end learning. We further show that with a suitable choice of message aggregation operators, our FGNN is also able to represent Max-Product belief propagation, providing a single family of architecture that can represent both Max and Sum-Product loopy belief propagation. Our extensive experimental evaluation on synthetic as well as real datasets demonstrates the potential of the proposed model.

Inference in Probabilistic Graphical Models by Graph Neural Networks

Discovering the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions

Efficient Probabilistic Logic Reasoning with Graph Neural Networks

Rethinking and Extending the Probabilistic Inference Capacity of GNNs

Neuralizing Efficient Higher-order Belief Propagation

Can Graph Neural Networks Help Logic Reasoning?

A comparison of algorithms for inference and learning in probabilistic graphical models

A Study of Joint Graph Inference and Forecasting

Probabilistic Inference of Bayesian Neural Networks with Generalized Expectation Propagation

Factor Graph Neural Networks

Probability Passing for Graph Neural Networks: Graph Structure and Representations Joint Learning

On Neural Networks as Infinite Tree-Structured Probabilistic Graphical Models

Learning Dynamics and Structure of Complex Systems Using Graph Neural Networks

Neural Enhanced Belief Propagation on Factor Graphs

Graph Belief Propagation Networks

Inference Graphs for CNN Interpretation

Learning General Latent-Variable Graphical Models with Predictive Belief Propagation

Non-Parametric Graph Learning for Bayesian Graph Neural Networks

Hierarchical Graph Neural Network Based-on Semi-implicit Variational Inference

LLMExplainer: Large Language Model based Bayesian Inference for Graph Explanation Generation

Inference for Network Structure and Dynamics from Time Series Data via Graph Neural Network