Abstract:Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

ISOLATION OF ACETOIN-PRODUCING Bacillus STRAINS FROM JAPANESE TRADITIONAL FOOD—NATTO

Probabilistic Computation with Emerging Covariance: Towards Efficient Uncertainty Quantification

Bivariate Distribution Regression with Application to Insurance Data

Bivariate cubic normal distribution for non-Gaussian problems

Assessing Bayes Factor Surfaces Using Interactive Visualization and Computer Surrogate Modeling

Learning about probabilistic inference and forecasting by playing with multivariate normal distributions

Fitting Bell Curves to Data Distributions using Visualization

Dynamic Risk Prediction via a Joint Frailty-Copula Model and IPD Meta-Analysis: Building Web Applications

Evaluating Bayesian Model Visualisations

Visual Analysis of Large Multivariate Scattered Data using Clustering and Probabilistic Summaries

Central Limit Theorem: New SOCR Applet and Demonstration Activity

The Unit Folded Normal Distribution: A New Unit Probability Distribution with the Estimation Procedures, Quantile Regression Modeling and Educational Attainment Applications

Computing Marginal and Conditional Divergences between Decomposable Models with Applications

Biology-inspired geometric representation of probability and applications to completion and options' pricing

A Tractable Probability Distribution with Applications in Three-Dimensional Statistics

Design and analysis of computer experiments with both numeral and distribution inputs

Nonparametric forecasting of multivariate probability density functions

Probability distributions and calculations for Hake's ratio statistics in measuring effect size

BAMBI: An R package for Fitting Bivariate Angular Mixture Models

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Information-theoretic evaluation of covariate distributions models