Abstract:Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

Efficient multivariate entropy estimation via $k$-nearest neighbour distances

Contribution to Transfer Entropy Estimation Via the K-Nearest-neighbors Approach

Empirical estimation of entropy functionals with confidence

An Efficient Integrated Nonparametric Entropy Estimator of Serial Dependence

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Measures of Entropy From Data Using Infinitely Divisible Kernels

Nearest-Neighbours Estimators for Conditional Mutual Information

Entropy Estimation via Uniformization

Asymptotic Distribution Theory for Nonparametric Entropy Measures of Serial Dependence

Improve KNN algorithm based on entropy method

Efficient Estimation of k for the Nearest Neighbors Class of Methods

Insights into Entropy As a Measure of Multivariate Variability.

Generating Multivariate Ordinal Data via Entropy Principles

Minimax rates of entropy estimation on large alphabets via best polynomial approximation

Random Forests for Adaptive Nearest Neighbor Estimation of Information-Theoretic Quantities

Varentropy Estimation via Nearest Neighbor Graphs

Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information

Neural Joint Entropy Estimation

On the Efficient Estimation of Min-Entropy

From ε-entropy to KL-entropy: Analysis of minimum information complexity density estimation

Distance-based and RKHS-based dependence metrics in high dimension