Information-Theoretic Measures on Lattices for High-Order Interactions
Zhaolu Liu,Mauricio Barahona,Robert L. Peach
2024-10-11
Abstract:Traditional models based solely on pairwise associations often fail to capture the complex statistical structure of multivariate data. Existing approaches for identifying information shared among groups of $d>3$ variables are frequently computationally intractable, asymmetric with respect to a target variable, or unable to account for all factorisations of the joint probability distribution. We present a systematic framework that derives higher-order information-theoretic measures using lattice and operator function pairs, whereby the lattice representing the algebraic relationships among variables, with operator functions that compute the measures over the lattice. We show that many commonly used measures can be derived within this framework, however they are often restricted to sublattices of the partition lattice, which prevents them from capturing all interactions when $d>3$. We also demonstrate that KL divergence, when used as an operator function, leads to unwanted cancellation of interactions for $d>3$. To fully characterise all interactions among $d$ variables, we introduce the Streitberg Information, using generalisations of KL divergence as an operator function, and defined over the full partition lattice. We validate Streitberg Information numerically on synthetic data, and illustrate its application in analysing complex interactions among stocks, decoding neural signals, and performing feature selection in machine learning.
Information Theory,Machine Learning