Abstract:Finding the model that best describes a high-dimensional dataset is a daunting task, even more so if one aims to consider all possible high-order patterns of the data, going beyond pairwise models. For binary data, we show that this task becomes feasible when restricting the search to a family of simple models, that we call Minimally Complex Models (MCMs). MCMs are maximum entropy models that have interactions of arbitrarily high order grouped into independent components of minimal complexity. They are simple in information-theoretic terms, which means they can only fit well certain types of data patterns and are therefore easy to falsify. We show that Bayesian model selection restricted to these models is computationally feasible and has many advantages. First, the model evidence, which balances goodness-of-fit against complexity, can be computed efficiently without any parameter fitting, enabling very fast explorations of the space of MCMs. Second, the family of MCMs is invariant under gauge transformations, which can be used to develop a representation-independent approach to statistical modeling. For small systems (up to 15 variables), combining these two results allows us to select the best MCM among all, even though the number of models is already extremely large. For larger systems, we propose simple heuristics to find optimal MCMs in reasonable times. Besides, inference and sampling can be performed without any computational effort. Finally, because MCMs have interactions of any order, they can reveal the presence of important high-order dependencies in the data, providing a new approach to explore high-order dependencies in complex systems. We apply our method to synthetic data and real-world examples, illustrating how MCMs portray the structure of dependencies among variables in a simple manner, extracting falsifiable predictions on symmetries and invariance from the data.

A novel low-complexity method for determining nonadditive interaction measures based on least-norm learning

Relative Synergy Coefficient: A Novel Way to Detect Variable Interaction in Large Dataset

Precise determination of non-additive measures

Interaction Measures, Partition Lattices and Kernel Tests for High-Order Interactions

Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension

Significance Measure with Nonlinear and Incommensurable Observations

Nonparametric Estimation and Testing of Interaction in Additive Models

Optimal minimax rate of learning interaction kernels

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Predicting pairwise interaction affinities with l0-penalized least squares–a nonsmooth bi-objective optimization based approach*

Learning Theory for Inferring Interaction Kernels in Second-Order Interacting Agent Systems

High-dimensional Interactions Detection with Sparse Principal Hessian Matrix

On efficient dimension reduction with respect to the interaction between two response variables

The monotonic linear model: Testing for removable interactions.

Bayesian Inference of Minimally Complex Models with Interactions of Arbitrary Order

Symphony in the Latent Space: Provably Integrating High-dimensional Techniques with Non-linear Machine Learning Models

Inference of interaction kernels in mean-field models of opinion dynamics

Interacting Particle Systems on Networks: joint inference of the network and the interaction kernel

Information-Theoretic Measures on Lattices for High-Order Interactions

On nonparametric estimation of the interaction function in particle system models

A Theoretical Framework for Interaction Measure and Sensitivity Analysis in Cross-Layer Design