Abstract:Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

Information Characteristics of Discrete K-L Transform Based on Information Entropy

Discrete K-L Transform And Information Feature Analysis

K-L Information Rate and H/sub ∞/ Entropy in Linear Control Systems

Nonequilibrium in Thermodynamic Formalism: the Second Law, gases and Information Geometry

On Mutual Information, Mmse And H-Infinity Entropy In Stochastic Linear Feedback Systems

Information theoretic interpretations for H˞ entropy

Δ-Entropy: Definition, Properties and Applications in System Identification with Quantized Data

An Information Theoretic Interpretation for H-Infinity Entropy

Measures of Entropy From Data Using Infinitely Divisible Kernels

The Duality of Entropy/Extropy, and Completion of the Kullback Information Complex

Generalized information entropy and generalized information dimension

Reflections on Shannon Information: In search of a natural information-entropy for images

Lossless Information Hiding In Images On Transform Domains

Fractal Measure Theory for Knowledge Representation

On the Structure of Information

The Basic Discrete Hilbert Transform with an Information Hiding Application

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Relations between entropy rate, entropy production and information geometry in linear stochastic systems

Psychological Information Entropy and Its Applications to Reliability Engineering

An Information-Theoretic Analysis of Discrete-Time Control and Filtering Limitations by the I-MMSE Relationships

On Thermodynamic Information