Abstract:Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

Discrete K-L Transform And Information Feature Analysis

Information Characteristics of Discrete K-L Transform Based on Information Entropy

Data-Dependent Kernel Discriminant Analysis for Feature Extraction and Classification

Information Feature Compression Algorithm Based on SCEC

Information Feature Analysis and Improved Algorithm of PCA

Symmetric cross entropy and information feature compression algorithm

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Discretization Algorithm Based on Effective Information Ratio

A novel discretization algorithm based on multi-scale and information entropy

An Advanced Hyperspectral Band Selection Approach Based on Mutual Information.

Measures of Entropy From Data Using Infinitely Divisible Kernels

Discrete Social Recommendation.

A Novel Pca Improved Algorithm

Robust and Fast Measure of Information Via Low-Rank Representation

Supervised Feature Extraction Algorithm Based on Improved Polynomial Entropy

An improved PCA algorithm based on WIF.

A Novel Supervised Information Feature Compression Algorithm

The Discrete Cosine Transform and Its Impact on Visual Compression: Fifty Years From Its Invention

Efficient Statistics: Extracting Information from IID Observations

Supervised Information Feature Compression Algorithm Based on Divergence Criterion

An Interpretable Compression and Classification System: Theory and Applications