Some asymptotic behaviors for the plug-in estimator of entropy

Zhenhong Yu,Yu Miao

2024-09-28

Abstract:In the present paper, we consider the plug-in estimator of Shannon's entropy defined on a finite alphabet which is assumed to dynamically vary as the sample size increases. The asymptotic behaviors for the plug-in estimator, such as, asymptotic normality, Berry-Esseen bound and moderate deviation principle, are established.

Probability

What problem does this paper attempt to address?

Asymptotic Normality for Plug-in Estimators of Generalized Shannon's Entropy

Jialin Zhang,Jingyi Shi

DOI: https://doi.org/10.3390/e24050683

2022-05-03

Abstract:Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon's entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon's entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon's entropy. The asymptotic distribution for Shannon's entropy's plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon's entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow interval estimation and statistical tests with generalized Shannon's entropy.

Statistics Theory,Information Theory
Asymptotic analysis of random partitions ∗

Z. Su

2006-01-01

Abstract:In this paper we aim to review some works on the asymptotic behaviors of random partitions. The focus is upon two basic probability models— uniform partitions and Plancherel partitions. One of fundamental results is the existence of limit shapes, which corresponds to the classic law of large numbers. The fluctuations around the limit shape are also considered; the well-known Gumbel distribution, Tracy-Widom distribution and normal distribution will be used to describe the asymptotic fluctuations at the edge and in the bulk of the spectrum of random partitions. The technique used in asymptotic analysis is the so-called poissonization and depoissonization method, a peculiar conditioning argument. We are content with describing some basic facts and remarkable results; no complete proof is given.
Consistency of the Plug-In Estimator of the Entropy Rate for Ergodic Processes

Łukasz Dębowski

DOI: https://doi.org/10.1109/ISIT.2016.7541579

2016-04-26

Abstract:A plug-in estimator of entropy is the entropy of the distribution where probabilities of symbols or blocks have been replaced with their relative frequencies in the sample. Consistency and asymptotic unbiasedness of the plug-in estimator can be easily demonstrated in the IID case. In this paper, we ask whether the plug-in estimator can be used for consistent estimation of the entropy rate $h$ of a stationary ergodic process. The answer is positive if, to estimate block entropy of order $k$, we use a sample longer than $2^{k(h+\epsilon)}$, whereas it is negative if we use a sample shorter than $2^{k(h-\epsilon)}$. In particular, if we do not know the entropy rate $h$, it is sufficient to use a sample of length $(|X|+\epsilon)^{k}$ where $|X|$ is the alphabet size. The result is derived using $k$-block coding. As a by-product of our technique, we also show that the block entropy of a stationary process is bounded above by a nonlinear function of the average block entropy of its ergodic components. This inequality can be used for an alternative proof of the known fact that the entropy rate a stationary process equals the average entropy rate of its ergodic components.

Information Theory
Asymptotic Distributions of Innovation Density Estimators in Linear Processes

Xinmei Shen,Zhengyan Lin,Caiya Zhang

DOI: https://doi.org/10.1080/03610920701877586

2008-01-01

Abstract:In this article, we demonstrate that at a fixed point, the asymptotic distribution of the innovation density estimator is normal for stationary linear process. Also, we show that the asymptotic distribution of the global measure of the deviation of the density estimator from the expectation of the kernel innovation density (based on the true innovations) is the same as that in the case when we can observe the true innovations.
On Asymptotic Properties of the Parameter Estimator for a Type of SPDE

Caiya Zhang,Zhengyan Lin

DOI: https://doi.org/10.1016/j.jspi.2008.02.016

IF: 1.095

2008-01-01

Journal of Statistical Planning and Inference

Abstract:In this paper, we study a random field Uɛ(t,x) governed by some type of stochastic partial differential equations with an unknown parameter θ and a small noise ɛ. We construct an estimator of θ based on the continuous observation of N Fourier coefficients of Uɛ(t,x), and prove the strong convergence and asymptotic normality of the estimator when the noise ɛ tends to zero.
The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

Wei-Dong Liu,Zhengyan Lin,Qi-Man Shao

DOI: https://doi.org/10.1214/08-aap527

2008-01-01

Abstract:Let X-1 , . . . , X-n be a random sample from a p-dimensional population distribution. Assume that c(1)n(alpha) <= p <= c(2)n(alpha) for some positive constants c(1), c(2) and alpha. In this paper we introduce a new statistic for testing independence of the p-variates of the population and prove that the limiting distribution is the extreme distribution of type I with a rate of convergence O ((log n)(5/2)/root n). This is much faster than O (1/log n), a typical convergence rate for this type of extreme distribution. A simulation study and application to stochastic optimization are discussed.
Asymptotic Distribution Theory for Nonparametric Entropy Measures of Serial Dependence

YM Hong,H White

DOI: https://doi.org/10.1111/j.1468-0262.2005.00597.x

IF: 6.1

2005-01-01

Econometrica

Abstract:Entropy is a classical statistical concept with appealing properties. Establishing asymptotic distribution theory for smoothed nonparametric entropy measures of dependence has so far proved challenging. In this paper, we develop an asymptotic theory for a class of kernel-based smoothed nonparametric entropy measures of serial dependence in a time-series context. We use this theory to derive the limiting distribution of Granger and Lin's (1994) normalized entropy measure of serial dependence, which was previously not available in the literature. We also apply our theory to construct a new entropy-based test for serial dependence, providing an alternative to Robinson's (1991) approach. To obtain accurate inferences, we propose and justify a consistent smoothed bootstrap procedure. The naive bootstrap is not consistent for our test. Our test is useful in, for example, testing the random walk hypothesis, evaluating density forecasts, and identifying important lags of a time series. It is asymptotically locally more powerful than Robinson's (1991) test, as is confirmed in our simulation. An application to the daily S&P 500 stock price index illustrates our approach.
Asymptotic Properties of Nonparametric M-Estimation for Mixing Functional Data

Jia Chen,Lixin Zhang

DOI: https://doi.org/10.1016/j.jspi.2008.05.007

IF: 1.095

2009-01-01

Journal of Statistical Planning and Inference

Abstract:We investigate the asymptotic behavior of a nonparametric M-estimator of a regression function for stationary dependent processes, where the explanatory variables take values in some abstract functional space. Under some regularity conditions, we give the weak and strong consistency of the estimator as well as its asymptotic normality. We also give two examples of functional processes that satisfy the mixing conditions assumed in this paper. Furthermore, a simulated example is presented to examine the finite sample performance of the proposed estimator.
An Efficient Integrated Nonparametric Entropy Estimator of Serial Dependence

Yongmiao Hong,Xia Wang,Wenjie Zhang,Shouyang Wang

DOI: https://doi.org/10.1080/07474938.2017.1307564

IF: 1.2

2017-01-01

Econometric Reviews

Abstract:We propose an efficient numerical integration-based nonparametric entropy estimator for serial dependence and show that the new entropy estimator has a smaller asymptotic variance than Hong and White's (2005) sample average-based estimator. This delivers an asymptotically more efficient test for serial dependence. In particular, the uniform kernel gives the smallest asymptotic variance for the numerical integration-based entropy estimator over a class of positive kernel functions. Moreover, the naive bootstrap can be used to obtain accurate inferences for our test, whereas it is not applicable to Hong and White's (2005) sample averaging approach. A simulation study confirms the merits of our approach.
Assessment of Time Series Complexity Using Improved Approximate Entropy

Kong De-Ren,Xie Hong-Bo

DOI: https://doi.org/10.1088/0256-307x/28/9/090502

2011-01-01

Chinese Physics Letters

Abstract:Approximate entropy (ApEn), a measure quantifying complexity and/or regularity, is believed to be an effective method of analyzing diverse settings. However, the similarity definition of vectors based on Heaviside function may cause some problems in the validity and accuracy of ApEn. To overcome the problems, an improved approximate entropy (iApEn) based on the sigmoid function is proposed. The performance of iApEn is tested on the independent identically distributed (IID) Gaussian noise, the MIX stochastic model, the Rossler map, the logistic map, and the high-dimensional Mackey-Glass oscillator. The results show that iApEn is superior to ApEn in several aspects, including better relative consistency, freedom of parameter selection, robust to noise, and more independence on record length when characterizing time series with different complexities.
Berry–Esseen Bounds for Kernel Estimates of Stationaryprocesses

Chu Huang,Hanchao Wang,Lixin Zhang

DOI: https://doi.org/10.1016/j.jspi.2010.09.029

IF: 1.095

2010-01-01

Journal of Statistical Planning and Inference

Abstract:In this paper we obtain nonuniform Berry–Esseen bounds of the kernel estimate of stationary process {Xi} by the method of martingale approximation. In particular, by choice of bandwidth bn, the Berry–Esseen bounds can be n−2/15. The dependence condition of {Xi} is expressed in terms of physical dependence measures introduced by Wu (2005).
A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems

Assaf Pinchas,Irad Ben-Gal,Amichai Painsky

DOI: https://doi.org/10.3390/e26050369

IF: 2.738

2024-04-29

Entropy

Abstract:This paper presents a comparative study of entropy estimation in a large-alphabet regime. A variety of entropy estimators have been proposed over the years, where each estimator is designed for a different setup with its own strengths and caveats. As a consequence, no estimator is known to be universally better than the others. This work addresses this gap by comparing twenty-one entropy estimators in the studied regime, starting with the simplest plug-in estimator and leading up to the most recent neural network-based and polynomial approximate estimators. Our findings show that the estimators' performance highly depends on the underlying distribution. Specifically, we distinguish between three types of distributions, ranging from uniform to degenerate distributions. For each class of distribution, we recommend the most suitable estimator. Further, we propose a sample-dependent approach, which again considers three classes of distribution, and report the top-performing estimators in each class. This approach provides a data-dependent framework for choosing the desired estimator in practical setups.

physics, multidisciplinary
Asymptotic Normality of the Recursive M-estimators of the Scale Parameters

Baiqi Miao,Yuehua Wu,Donghai Liu,Qian Tong

DOI: https://doi.org/10.1007/s10463-006-0051-y

2006-01-01

Annals of the Institute of Statistical Mathematics

Abstract:In this paper, the limit distributions of the recursive M-estimators of scatter parameters in a multivariate linear model setting are studied. Under some mild conditions, the asymptotic normality of the recursive M-esimtators is established. Some Monte Carlo simulation results are presented to illustrate the performance of the recursive M-estimators.
Empirical estimation of entropy functionals with confidence

Kumar Sricharan,Raviv Raich,Alfred O. Hero III

DOI: https://doi.org/10.48550/arXiv.1012.4188

2012-02-26

Abstract:This paper introduces a class of k-nearest neighbor ($k$-NN) estimators called bipartite plug-in (BPI) estimators for estimating integrals of non-linear functions of a probability density, such as Shannon entropy and Rényi entropy. The density is assumed to be smooth, have bounded support, and be uniformly bounded from below on this set. Unlike previous $k$-NN estimators of non-linear density functionals, the proposed estimator uses data-splitting and boundary correction to achieve lower mean square error. Specifically, we assume that $T$ i.i.d. samples ${X}_i \in \mathbb{R}^d$ from the density are split into two pieces of cardinality $M$ and $N$ respectively, with $M$ samples used for computing a k-nearest-neighbor density estimate and the remaining $N$ samples used for empirical estimation of the integral of the density functional. By studying the statistical properties of k-NN balls, explicit rates for the bias and variance of the BPI estimator are derived in terms of the sample size, the dimension of the samples and the underlying probability distribution. Based on these results, it is possible to specify optimal choice of tuning parameters $M/T$, $k$ for maximizing the rate of decrease of the mean square error (MSE). The resultant optimized BPI estimator converges faster and achieves lower mean squared error than previous $k$-NN entropy estimators. In addition, a central limit theorem is established for the BPI estimator that allows us to specify tight asymptotic confidence intervals.

Statistics Theory,Machine Learning
Minimax rates of entropy estimation on large alphabets via best polynomial approximation

Yihong Wu,Pengkun Yang

DOI: https://doi.org/10.48550/arXiv.1407.0381

2016-02-18

Abstract:Consider the problem of estimating the Shannon entropy of a distribution over $k$ elements from $n$ independent samples. We show that the minimax mean-square error is within universal multiplicative constant factors of $$\Big(\frac{k }{n \log k}\Big)^2 + \frac{\log^2 k}{n}$$ if $n$ exceeds a constant factor of $\frac{k}{\log k}$; otherwise there exists no consistent estimator. This refines the recent result of Valiant-Valiant \cite{VV11} that the minimal sample size for consistent entropy estimation scales according to $\Theta(\frac{k}{\log k})$. The apparatus of best polynomial approximation plays a key role in both the construction of optimal estimators and, via a duality argument, the minimax lower bound.

Information Theory,Statistics Theory
Asymptotic Approximation to the Entropy Rate of Binary Hidden Markov Processes

CHEN Shuang-ping,ZHENG Hao-ran,TONG Qing,WANG Xu-fa

DOI: https://doi.org/10.3321/j.issn:0372-2112.2006.03.026

2006-01-01

Abstract:Based on the convergence of bounds for the entropy rate of binary hidden Markov processes,a numerical approach is advanced.The algorithm can approximate to true value of the entropy rate below a predefined error,and the accuracy can also be estimated.Since the logarithm of the algorithm′s complexity is linear to the logarithm of error,the cost of the algorithm is acceptable for practical use in engineering fields.It casts light to solve the problem of computing the entropy rate of more generalized kinds of HMPs.
Some new asymptotic results for series estimation under clustered dependence

Pingbo Hu,Xiuyuan Peng,Xinglin Hu

DOI: https://doi.org/10.1016/j.spl.2024.110156

IF: 0.718

2024-05-31

Statistics & Probability Letters

Abstract:This paper presents some asymptotic results for series estimation of a nonparametric regression model under clustered dependence. A mean square rate of convergence for the series regression estimator is established. Moreover, asymptotic pointwise normality is shown for the series estimator.

statistics & probability
A Scalable Shannon Entropy Estimator

Priyanka Golia,Brendan Juba,Kuldeep S. Meel

DOI: https://doi.org/10.48550/arXiv.2206.00921

2022-06-02

Cryptography and Security

Abstract:We revisit the well-studied problem of estimating the Shannon entropy of a probability distribution, now given access to a probability-revealing conditional sampling oracle. In this model, the oracle takes as input the representation of a set $S$ and returns a sample from the distribution obtained by conditioning on $S$, together with the probability of that sample in the distribution. Our work is motivated by applications of such algorithms in Quantitative Information Flow analysis (QIF) in programming-language-based security. Here, information-theoretic quantities capture the effort required on the part of an adversary to obtain access to confidential information. These applications demand accurate measurements when the entropy is small. Existing algorithms that do not use conditional samples require a number of queries that scale inversely with the entropy, which is unacceptable in this regime, and indeed, a lower bound by Batu et al.(STOC 2002) established that no algorithm using only sampling and evaluation oracles can obtain acceptable performance. On the other hand, prior work in the conditional sampling model by Chakraborty et al.(SICOMP 2016) only obtained a high-order polynomial query complexity, $\mathcal{O}(\frac{m^7}{\epsilon^8}\log\frac{1}{\delta})$ queries, to obtain additive $\epsilon$-approximations on a domain of size $\mathcal{O}(2^m)$. We obtain multiplicative $(1+\epsilon)$-approximations using only $\mathcal{O}(\frac{m}{\epsilon^2}\log\frac{1}{\delta})$ queries to the probability-revealing conditional sampling oracle. Indeed, moreover, we obtain small, explicit constants, and demonstrate that our algorithm obtains a substantial improvement in practice over the previous state-of-the-art methods used for entropy estimation in QIF.
Neural Joint Entropy Estimation

Yuval Shalev,Amichai Painsky,Irad Ben-Gal

DOI: https://doi.org/10.1109/tnnls.2022.3204919

IF: 14.255

2022-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics, and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos. The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information (MI). We show that these estimators are strongly consistent and demonstrate their performance in a variety of use cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to MI estimation. Next, we apply the proposed scheme to conditional MI estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy (TE) estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all of these setups.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Entropies and their Asymptotic Theory in the discrete case

Amadou Diadie Ba,Gane Samb Lo

DOI: https://doi.org/10.48550/arXiv.1903.08645

2019-03-20

Abstract:We present some new nonparametric estimators of entropies and we establish almost sure consistency and central limit Theorems for some of the most important entropies in the discrete case. Our theorical results are validated by simulations.

Statistics Theory

Some asymptotic behaviors for the plug-in estimator of entropy

Asymptotic Normality for Plug-in Estimators of Generalized Shannon's Entropy

Asymptotic analysis of random partitions ∗

Consistency of the Plug-In Estimator of the Entropy Rate for Ergodic Processes

Asymptotic Distributions of Innovation Density Estimators in Linear Processes

On Asymptotic Properties of the Parameter Estimator for a Type of SPDE

The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

Asymptotic Distribution Theory for Nonparametric Entropy Measures of Serial Dependence

Asymptotic Properties of Nonparametric M-Estimation for Mixing Functional Data

An Efficient Integrated Nonparametric Entropy Estimator of Serial Dependence

Assessment of Time Series Complexity Using Improved Approximate Entropy

Berry–Esseen Bounds for Kernel Estimates of Stationaryprocesses

A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems

Asymptotic Normality of the Recursive M-estimators of the Scale Parameters

Empirical estimation of entropy functionals with confidence

Minimax rates of entropy estimation on large alphabets via best polynomial approximation

Asymptotic Approximation to the Entropy Rate of Binary Hidden Markov Processes

Some new asymptotic results for series estimation under clustered dependence

A Scalable Shannon Entropy Estimator

Neural Joint Entropy Estimation

Entropies and their Asymptotic Theory in the discrete case