Abstract:Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

Estimators of Entropy and Information via Inference in Probabilistic Models

Efficient Estimation of the Value of Information in Monte Carlo Models

Measures of Entropy From Data Using Infinitely Divisible Kernels

Uncertainty and Value of Information in Risk Prediction Modeling

Neural Joint Entropy Estimation

Insights into the Robustness of Minimum Error Entropy Estimation

Mutual Information Estimation via $f$-Divergence and Data Derangements

Convergence of Statistical Estimators via Mutual Information Bounds

Inferential Moments of Uncertain Multivariable Systems

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Variational Inference: A Review for Statisticians

Generalizing Information to the Evolution of Rational Belief

New information-estimation results for poisson, binomial and negative binomial models

Variational Entropy Search for Adjusting Expected Improvement

Improving Mutual Information Estimation with Annealed and Energy-Based Bounds

REMEDI: Corrective Transformations for Improved Neural Entropy Estimation

Generalizing Geometric Partition Entropy for the Estimation of Mutual Information in the Presence of Informative Outliers

Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

Generalized Information Theory Meets Human Cognition: Introducing a Unified Framework to Model Uncertainty and Information Search

Information Theoretic Lower Bounds for Information Theoretic Upper Bounds

On Variational Bounds of Mutual Information