Abstract:Finding the model that best describes a high-dimensional dataset is a daunting task, even more so if one aims to consider all possible high-order patterns of the data, going beyond pairwise models. For binary data, we show that this task becomes feasible when restricting the search to a family of simple models, that we call Minimally Complex Models (MCMs). MCMs are maximum entropy models that have interactions of arbitrarily high order grouped into independent components of minimal complexity. They are simple in information-theoretic terms, which means they can only fit well certain types of data patterns and are therefore easy to falsify. We show that Bayesian model selection restricted to these models is computationally feasible and has many advantages. First, the model evidence, which balances goodness-of-fit against complexity, can be computed efficiently without any parameter fitting, enabling very fast explorations of the space of MCMs. Second, the family of MCMs is invariant under gauge transformations, which can be used to develop a representation-independent approach to statistical modeling. For small systems (up to 15 variables), combining these two results allows us to select the best MCM among all, even though the number of models is already extremely large. For larger systems, we propose simple heuristics to find optimal MCMs in reasonable times. Besides, inference and sampling can be performed without any computational effort. Finally, because MCMs have interactions of any order, they can reveal the presence of important high-order dependencies in the data, providing a new approach to explore high-order dependencies in complex systems. We apply our method to synthetic data and real-world examples, illustrating how MCMs portray the structure of dependencies among variables in a simple manner, extracting falsifiable predictions on symmetries and invariance from the data.

Classifying one-dimensional discrete models with maximum likelihood degree one

Matroid Stratification of ML Degrees of Independence Models

Varieties with maximum likelihood degree one

Bounding the maximum likelihood degree

Bayesian Inference of Minimally Complex Models with Interactions of Arbitrary Order

Homaloidal Polynomials and Gaussian Models of Maximum Likelihood Degree One

On the maximum likelihood degree for Gaussian graphical models

Lawrence Lifts, Matroids, and Maximum Likelihood Degrees

Maximum likelihood degree of the two-dimensional linear Gaussian covariance model

The classification of Boolean degree $1$ functions in high-dimensional finite vector spaces

Near Universal Consistency of the Maximum Pseudolikelihood Estimator for Discrete Models

Likelihood Geometry of Reflexive Polytopes

Statistical Inference for Generative Models with Maximum Mean Discrepancy

A delineation of new classes of exponential dispersion models supported on the set of nonnegative integers

Likelihood-free inference via classification

Mixtures of Discrete Decomposable Graphical Models

Nonparametric maximum likelihood estimation under a likelihood ratio order

Degree counts in random simplicial complexes of the preferential attachment type

Optimal errors and phase transitions in high-dimensional generalized linear models

Model-free Estimation of Latent Structure via Multiscale Nonparametric Maximum Likelihood

Identifiability and dependence structure in factor models through monomial assignments in determinantal decompositions