Abstract:Dependency networks (Heckerman et al., 2000) are potential probabilistic graphical models for systems comprising a large number of variables. Like Bayesian networks, the structure of a dependency network is represented by a directed graph, and each node has a conditional probability table. Learning and inference are realized locally on individual nodes; therefore, computation remains tractable even with a large number of variables. However, the dependency network's learned distribution is the stationary distribution of a Markov chain called pseudo-Gibbs sampling and has no closed-form expressions. This technical disadvantage has impeded the development of dependency networks. In this paper, we consider a certain manifold for each node. Then, we can interpret pseudo-Gibbs sampling as iterative m-projections onto these manifolds. This interpretation provides a theoretical bound for the location where the stationary distribution of pseudo-Gibbs sampling exists in distribution space. Furthermore, this interpretation involves structure and parameter learning algorithms as optimization problems. In addition, we compare dependency and Bayesian networks experimentally. The results demonstrate that the dependency network and the Bayesian network have roughly the same performance in terms of the accuracy of their learned distributions. The results also show that the dependency network can learn much faster than the Bayesian network.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the technical difficulties encountered in the learning and reasoning processes of Dependency Networks. Specifically, the learned distribution of the Dependency Networks is the stationary distribution of the pseudo - Gibbs sampling Markov chain and has no closed - form expression. This technical disadvantage has hindered the development of Dependency Networks. To solve this problem, the author re - considers the Dependency Networks from the perspective of information geometry and presents new insights. ### Summary of Main Problems: 1. **No Closed - Form for the Stationary Distribution of Dependency Networks**: The learned distribution of the Dependency Networks is the stationary distribution of the pseudo - Gibbs sampling, but this distribution has no closed - form expression, resulting in the inability to directly apply the traditional maximum - likelihood estimation method. 2. **Learning and Reasoning Efficiency of Dependency Networks**: Although the Dependency Networks can handle a large number of variables and have low computational complexity, due to the complexity of their stationary distribution, there are difficulties in theoretical analysis and practical applications. ### Solutions: 1. **Perspective of Information Geometry**: By introducing the method of information geometry, the author interprets the pseudo - Gibbs sampling as an iterative m - projection onto certain manifolds. This interpretation provides a theoretical boundary for the stationary distribution of the Dependency Networks in the distribution space. 2. **Structure and Parameter Learning Algorithms**: Based on the above interpretation, the author proposes a new structure and parameter learning algorithm, which is regarded as an optimization problem. These algorithms make the learning process of the Dependency Networks more efficient and accurate. 3. **Experimental Comparison**: The author experimentally compares the performance of the Dependency Networks and Bayesian Networks. The results show that the two are approximately the same in terms of the accuracy of distribution learning, but the Dependency Networks have a faster learning speed. ### Key Contributions: - Proposed a new method of interpreting the Dependency Networks from the perspective of information geometry. - Provided a theoretical boundary for the stationary distribution of the Dependency Networks. - Developed more efficient structure and parameter learning algorithms. - Experimentally verified the performance advantages of the Dependency Networks, especially in terms of learning speed. Through these improvements, the author aims to promote the further development and use of the Dependency Networks in practical applications.

Reconsidering Dependency Networks from an Information Geometry Perspective

A New Inference Framework For Dependency Networks

Dependence and Relevance: A probabilistic view

Efficient dependency models for some distributions

Probability Distribution of Dependency Distance

Exploiting the Implicit Independence Assumption for Learning Directed Graphical Models.

Neural Dependencies Emerging from Learning Massive Categories.

Multivariate dependence and genetic networks inference

Probabilistic similarity networks

Reliable and Efficient Inference of Bayesian Networks from Sparse Data by Statistical Learning Theory

High Dimensional Logistic Regression Under Network Dependence

Inferring the dependence graph density of binary graphical models in high dimension

Extremal Dependence Concepts

Post-clustering Inference under Dependency

Graphical Models for Preference and Utility

Latent Dependency Forest Models

Probabilistic graphs using coupled random variables

Pseudo-likelihood-based $M$-estimation of random graphs with dependent edges and parameter vectors of increasing dimension

Deep Dependency Networks and Advanced Inference Schemes for Multi-Label Classification

A modeling framework for detecting and leveraging node-level information in Bayesian network inference

Scalable Bayesian High-dimensional Local Dependence Learning