Abstract:Learning high-dimensional distributions is a significant challenge in machine learning and statistics. Classical research has mostly concentrated on asymptotic analysis of such data under suitable assumptions. While existing works [Bhattacharyya et al.: SICOMP 2023, Daskalakis et al.: STOC 2021, Choo et al.: ALT 2024] focus on discrete distributions, the current work addresses the tree structure learning problem for Gaussian distributions, providing efficient algorithms with solid theoretical guarantees. This is crucial as real-world distributions are often continuous and differ from the discrete scenarios studied in prior works. In this work, we design a conditional mutual information tester for Gaussian random variables that can test whether two Gaussian random variables are independent, or their conditional mutual information is at least $\varepsilon$, for some parameter $\varepsilon \in (0,1)$ using $\mathcal{O}(\varepsilon^{-1})$ samples which we show to be near-optimal. In contrast, an additive estimation would require $\Omega(\varepsilon^{-2})$ samples. Our upper bound technique uses linear regression on a pair of suitably transformed random variables. Importantly, we show that the chain rule of conditional mutual information continues to hold for the estimated (conditional) mutual information. As an application of such a mutual information tester, we give an efficient $\varepsilon$-approximate structure-learning algorithm for an $n$-variate Gaussian tree model that takes $\widetilde{\Theta}(n\varepsilon^{-1})$ samples which we again show to be near-optimal. In contrast, when the underlying Gaussian model is not known to be tree-structured, we show that $\widetilde{\Theta}(n^2\varepsilon^{-2})$ samples are necessary and sufficient to output an $\varepsilon$-approximate tree structure. We perform extensive experiments that corroborate our theoretical convergence bounds.

Chernoff Information of Bottleneck Gaussian Trees.

Chernoff Information Between Gaussian Trees

Topological and Algebraic Properties of Chernoff Information Between Gaussian Graphs

Generalized Information Bottleneck for Gaussian Variables

Efficient Sample-optimal Learning of Gaussian Tree Models via Sample-optimal Testing of Gaussian Mutual Information

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

CCGIB: A Cross-Channel Graph Information Bottleneck Principle

Gaussian Information Bottleneck and the Non-Perturbative Renormalization Group

A unified framework for information-theoretic generalization bounds

The Double-Sided Information Bottleneck Function

Informational Divergence and Entropy Rate on Rooted Trees with Probabilities

Bounds on the Bethe Free Energy for Gaussian Networks

Towards a More Stable and General Subgraph Information Bottleneck

Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Efficient and Provably Convergent Computation of Information Bottleneck: A Semi-Relaxed Approach

A Matrix Version of Chernoff Inequality

Optimal estimation of Gaussian (poly)trees

Computation-information gap in high-dimensional clustering

Information Cascades on Arbitrary Topologies

Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem

Chernoff fusion of Bernoulli Gaussian max filters