Abstract:The recently developed matrix-based Rényi's $\alpha$ -order entropy enables measurement of information in data simply using the eigenspectrum of symmetric positive semi-definite (PSD) matrices in reproducing kernel Hilbert space, without estimation of the underlying data distribution. This intriguing property makes this new information measurement widely adopted in multiple statistical inference and learning tasks. However, the computation of such quantity involves the trace operator on a PSD matrix $G$ to power $\alpha$ (i.e., ${\text{tr}}(G^\alpha)$ ), with a normal complexity of nearly $\mathcal {O}(n^{3})$ , which severely hampers its practical usage when the number of samples (i.e., $n$ ) is large. In this work, we present computationally efficient approximations to this new entropy functional that can reduce its complexity to even significantly less than $\mathcal {O}(n^{2})$ . To this end, we leverage the recent progress on Randomized Numerical Linear Algebra, developing Taylor, Chebyshev and Lanczos approximations to ${\text{tr}}(G^\alpha)$ for arbitrary values of $\alpha$ by converting it into a matrix-vector multiplication problem. We also establish the connection between the matrix-based Rényi's entropy and PSD matrix approximation, which enables exploiting both clustering and block low-rank structure of $G$ to further reduce the computational cost. We theoretically provide approximation accuracy guarantees and illustrate the properties for different approximations. Large-scale experimental evaluations on both synthetic and real-world data corroborate our theoretical findings, showing promising speedup with negligible loss in accuracy.

Deep Rényi Entropy Graph Kernel

A Graph Kernel From The Depth-Based Representation

Labeled Subgraph Entropy Kernel

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Rényi’s Entropy Perspective

A Mixed Entropy Local-Global Reproducing Kernel for Attributed Graphs.

Motif Entropy Graph Kernel

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective

A Hybrid Reproducing Graph Kernel Based on Information Entropy

On graph entropy measures based on the number of independent sets and matchings

Non-Hermitian Generalization of Rényi Entropy

Rényi entanglement entropy of spin chain with Generative Neural Networks

On the Rényi Cross-Entropy

Generalizing Körner's Graph Entropy to Graphons.

Novel Inequalities for Generalized Graph Entropies Revisited, Graph Energies and Topological Indices

Jensen-Shannon Graph Kernel Using Information Functionals

Generalizing K\"orner's graph entropy to graphons

Computationally Efficient Approximations for Matrix-Based Renyi's Entropy

Graph Kernels from the Jensen-Shannon Divergence

Entropy of Exchangeable Random Graphs

Connections Between Generalized Graph Entropies and Graph Energy

On the Von Neumann Entropy of Graphs