Abstract:The recently developed matrix-based Rényi's $\alpha$ -order entropy enables measurement of information in data simply using the eigenspectrum of symmetric positive semi-definite (PSD) matrices in reproducing kernel Hilbert space, without estimation of the underlying data distribution. This intriguing property makes this new information measurement widely adopted in multiple statistical inference and learning tasks. However, the computation of such quantity involves the trace operator on a PSD matrix $G$ to power $\alpha$ (i.e., ${\text{tr}}(G^\alpha)$ ), with a normal complexity of nearly $\mathcal {O}(n^{3})$ , which severely hampers its practical usage when the number of samples (i.e., $n$ ) is large. In this work, we present computationally efficient approximations to this new entropy functional that can reduce its complexity to even significantly less than $\mathcal {O}(n^{2})$ . To this end, we leverage the recent progress on Randomized Numerical Linear Algebra, developing Taylor, Chebyshev and Lanczos approximations to ${\text{tr}}(G^\alpha)$ for arbitrary values of $\alpha$ by converting it into a matrix-vector multiplication problem. We also establish the connection between the matrix-based Rényi's entropy and PSD matrix approximation, which enables exploiting both clustering and block low-rank structure of $G$ to further reduce the computational cost. We theoretically provide approximation accuracy guarantees and illustrate the properties for different approximations. Large-scale experimental evaluations on both synthetic and real-world data corroborate our theoretical findings, showing promising speedup with negligible loss in accuracy.

Fast calculation of entropy with Zhang's estimator

Bounds on Rényi and Shannon Entropies for Finite Mixtures of Multivariate Skew-Normal Distributions: Application to Swordfish (Xiphias gladius Linnaeus)

An Estimate Method Of The Minimum Entropy Of Natural Languages

A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems

Entropy Estimators for Markovian Sequences: A Comparative Analysis

Estimating the Minimum Entropy of Chinese and Japanese Languages.

Asymptotic Normality for Plug-in Estimators of Generalized Shannon's Entropy

Some asymptotic behaviors for the plug-in estimator of entropy

A Super Fast Algorithm for Estimating Sample Entropy.

Entropy of a Zipfian Distributed Lexicon

Entropy of Chinese and the perplexity of the language models

A Fast Algorithm for Computing Sample Entropy.

Determining the Number of Samples Required to Estimate Entropy in Natural Sequences

Computationally Efficient Approximations for Matrix-Based Renyi's Entropy

A New Look at the Classical Entropy of Written English

On the Entropy of Written Spanish

Entropy Estimation via Uniformization

A Quantum Algorithm Framework for Discrete Probability Distributions with Applications to Rényi Entropy Estimation

Neural Joint Entropy Estimation

On some entropy inequalities