Abstract:Learning high-dimensional distributions is a significant challenge in machine learning and statistics. Classical research has mostly concentrated on asymptotic analysis of such data under suitable assumptions. While existing works [Bhattacharyya et al.: SICOMP 2023, Daskalakis et al.: STOC 2021, Choo et al.: ALT 2024] focus on discrete distributions, the current work addresses the tree structure learning problem for Gaussian distributions, providing efficient algorithms with solid theoretical guarantees. This is crucial as real-world distributions are often continuous and differ from the discrete scenarios studied in prior works. In this work, we design a conditional mutual information tester for Gaussian random variables that can test whether two Gaussian random variables are independent, or their conditional mutual information is at least $\varepsilon$, for some parameter $\varepsilon \in (0,1)$ using $\mathcal{O}(\varepsilon^{-1})$ samples which we show to be near-optimal. In contrast, an additive estimation would require $\Omega(\varepsilon^{-2})$ samples. Our upper bound technique uses linear regression on a pair of suitably transformed random variables. Importantly, we show that the chain rule of conditional mutual information continues to hold for the estimated (conditional) mutual information. As an application of such a mutual information tester, we give an efficient $\varepsilon$-approximate structure-learning algorithm for an $n$-variate Gaussian tree model that takes $\widetilde{\Theta}(n\varepsilon^{-1})$ samples which we again show to be near-optimal. In contrast, when the underlying Gaussian model is not known to be tree-structured, we show that $\widetilde{\Theta}(n^2\varepsilon^{-2})$ samples are necessary and sufficient to output an $\varepsilon$-approximate tree structure. We perform extensive experiments that corroborate our theoretical convergence bounds.

Using Monte Carlo Tree Search to Calculate Mutual Information in High Dimensions

Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions

Beyond Normal: On the Evaluation of Mutual Information Estimators

Approximating mutual information of high-dimensional variables using learned representations

Latest CHORUS and NOMAD results

Mutual Information calculation on different appearances

Mutual Information Multinomial Estimation

Accurate estimation of the normalized mutual information of multidimensional data

Efficient Sample-optimal Learning of Gaussian Tree Models via Sample-optimal Testing of Gaussian Mutual Information

Feature selection based on Mutual Information in supervised learning

Fast Mutual Information Computation for Large Binary Datasets

A Unified Definition of Mutual Information with Applications in Machine Learning

Robust computation of mutual information using spatially adaptive meshes

Nearest-Neighbours Estimators for Conditional Mutual Information

Mutual information and the encoding of contingency tables

Learning non-parametric Markov networks with mutual information

Estimation and Confidence Intervals for Mutual Information: Issues in Convergence for Non-Normal Distributions

Estimating Total Correlation with Mutual Information Estimators

Random Forests for Adaptive Nearest Neighbor Estimation of Information-Theoretic Quantities

Multi-modality Medical Image Registration Using High Dimension Mutual Information

Investigating Bias in Non-Parametric Mutual Information Estimation.