Entropy and Probability Distributions

Y. Li,Vijay P. Singh,S. Cong

DOI: https://doi.org/10.1007/978-94-009-3953-0_26

1987-01-01

Abstract:Entropy was employed to investigate probability distribution functions and estimation of their parameters. Two curve-fitting methods, one of which is based on entropy, were compared using Monte Carlo simulation. A procedure to compare different distributions using entropy was suggested.

What problem does this paper attempt to address?

Properties of the Shannon, Rényi and other entropies: dependence in parameters, robustness in distributions and extremes

Iryna Bodnarchuk,Yuliya Mishura,Kostiantyn Ralchenko

2024-11-24

Abstract:We calculate and analyze various entropy measures and their properties for selected probability distributions. The entropies considered include Shannon, Rényi, generalized Rényi, Tsallis, Sharma-Mittal, and modified Shannon entropy, along with the Kullback-Leibler divergence. These measures are examined for several distributions, including gamma, chi-squared, exponential, Laplace, and log-normal distributions. We investigate the dependence of the entropy on the parameters of the respective distribution. We also study the convergence of Shannon entropy for certain probability distributions. Furthermore, we identify the extreme values of Shannon entropy for Gaussian vectors.

Information Theory,Probability
Entropy of function of uncertain variables.

Wei Dai,Xiaowei Chen

DOI: https://doi.org/10.1016/j.mcm.2011.08.052

2012-01-01

Mathematical and Computer Modelling

Abstract:Uncertainty theory is a branch of mathematics of studying the subjective uncertain behavior. Entropy is a key concept which provides a quantitative measurement of the uncertainty associated with uncertain variables. In order to compute the entropy more conveniently, this paper proposes some formulas of entropy of function of uncertain variables with regular uncertain distributions. Furthermore, the positive linearity property of the entropy operator is proved immediately following from these formulas at last.
Information-theoretic Methods for Estimating Complicated Probability Distributions

Zhi Zong

2006-01-01

Abstract:Preface Chapter 1. Randomness and probability Chapter 2. Inference and statistics Chapter 3. Random numbers and their applications Chapter 4. Approximation and B-spline function Chapter 5. Disorder, entropy and entropy estimation Chapter 6. Estimation of 1-D complicated distributions based on large samples Chapter 7. Estimation of 2-D complicated distributions based on large samples Chapter 8. Estimation of 1-D complicated distribution based on small samples Chapter 9. Estimation of 2-D complicated distribution based on small samples Chapter 10. Estimation of the membership function Chapter 11. Code specifications Bibliography Index
Sine Entropy for Uncertain Variables

Kai Yao,Jinwu Gao,Wei Dai

DOI: https://doi.org/10.1142/s0218488513500359

2013-01-01

Abstract:Entropy is a measure of the uncertainty associated with a variable whose value cannot be exactly predicated. In uncertainty theory, it has been quantified so far by logarithmic entropy. However, logarithmic entropy sometimes fails to measure the uncertainty. This paper will propose another type of entropy named sine entropy as a supplement, and explore its properties. After that, the maximum entropy principle will be introduced, and the arc-cosine distributed variables will be proved to have the maximum sine entropy with given expected value and variance.
Insights into Entropy As a Measure of Multivariate Variability.

Badong Chen,Jianji Wang,Haiquan Zhao,Jose C. Principe

DOI: https://doi.org/10.3390/e18050196

IF: 2.738

2016-01-01

Entropy

Abstract:Entropy has been widely employed as a measure of variability for problems, such as machine learning and signal processing. In this paper, we provide some new insights into the behaviors of entropy as a measure of multivariate variability. The relationships between multivariate entropy (joint or total marginal) and traditional measures of multivariate variability, such as total dispersion and generalized variance, are investigated. It is shown that for the jointly Gaussian case, the joint entropy (or entropy power) is equivalent to the generalized variance, while total marginal entropy is equivalent to the geometric mean of the marginal variances and total marginal entropy power is equivalent to the total dispersion. The smoothed multivariate entropy (joint or total marginal) and the kernel density estimation (KDE)-based entropy estimator (with finite samples) are also studied, which, under certain conditions, will be approximately equivalent to the total dispersion (or a total dispersion estimator), regardless of the data distribution.
Stochastic comparisons, differential entropy and varentropy for distributions induced by probability density functions

Antonio Di Crescenzo,Luca Paolillo,Alfonso Suárez-Llorens

DOI: https://doi.org/10.1007/s00184-024-00947-3

IF: 0.96

2024-02-10

Metrika

Abstract:Stimulated by the need of describing useful notions related to information measures, we introduce the 'pdf-related distributions'. These are defined in terms of transformation of absolutely continuous random variables through their own probability density functions. We investigate their main characteristics, with reference to the general form of the distribution, the quantiles, and some related notions of reliability theory. This allows us to obtain a characterization of the pdf-related distribution being uniform for distributions of exponential and Laplace type as well. We also face the problem of stochastic comparing the pdf-related distributions by resorting to suitable stochastic orders. Finally, the given results are used to analyse properties and to compare some useful information measures, such as the differential entropy and the varentropy.

statistics & probability
Entropy of Credibility Distributions for Fuzzy Variables

Pingke Li,Baoding Liu

DOI: https://doi.org/10.1109/TFUZZ.2007.894975

IF: 12.253

2008-01-01

IEEE Transactions on Fuzzy Systems

Abstract:This paper deals with the degree of uncertainty associated with fuzzy variables. Based on the notion of credibility measure, a definition of entropy is formulated from an information theoretical view and its properties are investigated. Finally, some comments are given on the construction of alternative definitions of entropy and axiomatic characterization.
Entropy methods in random motion

Piotr Garbaczewski

DOI: https://doi.org/10.48550/arXiv.cond-mat/0510533

2005-10-20

Statistical Mechanics

Abstract:We analyze a contrasting dynamical behavior of Gibbs-Shannon and conditional Kullback-Leibler entropies, induced by time-evolution of continuous probability distributions. The question of predominantly purpose-dependent entropy definition for non-equilibrium model systems is addressed. The conditional Kullback-Leibler entropy is often believed to properly capture physical features of an asymptotic approach towards equilibrium. We give arguments in favor of the usefulness of the standard Gibbs-type entropy and indicate that its dynamics gives an insight into physically relevant, but generally ignored in the literature, non-equilibrium phenomena. The role of physical units in the Gibbs-Shannon entropy definition is iscussed.
Entropy generating function for past lifetime and its properties

Smitha S.,Sudheesh K Kattumannil

2023-11-20

Abstract:The past entropy is considered as an uncertainty measure for the past lifetime distribution. Generating function approach to entropy become popular in recent time as it generate several well-known entropy measures. In this paper, we introduce the past entropy-generating function. We study certain properties of this measure. It is shown that the past entropy-generating function uniquely determines the distribution. Further, we present characterizations for some lifetime models using the relationship between reliability concepts and the past entropy-generating function.

Information Theory,Statistics Theory,Methodology
Exponential entropy of uncertain sets and its applications to learning curve and portfolio optimization

Chongguo Wang,Gang Shi,Yuhong Sheng,Hamed Ahmadzade

DOI: https://doi.org/10.3934/jimo.2024134

2024-10-01

Journal of Industrial and Management Optimization

Abstract:Exponential entropy is a concept in information theory that describes the amount of uncertainty or disorder in a system. It is based on the idea that as the number of possible states or configurations of a system increases exponentially, the entropy also increases exponentially. This means that as the complexity of a system grows, the amount of uncertainty about its state also grows rapidly. Exponential entropy is a key factor in understanding the behavior of complex systems and can help us predict how they will evolve over time. A formula has been derived for calculating the exponential entropy of uncertain sets through the inversion of membership functions. Additionally, by treating exponential entropy as a risk measure, we optimize portfolio selection problems using mean-entropy models.

engineering, multidisciplinary,operations research & management science,mathematics, interdisciplinary applications
Computing the Entropy Rate of Information Source with Methods of Statistical Physics

Shuang-ping Chen,Hao-ran Zheng,Meng Ma,Zhen-ya Zhang,Xu-fa Wang

2007-01-01

Abstract:From the mathematical point of view, information sources can be 1-to-1 mapped to stochastic processes. Known from the theory of chaos, multi-fractal of stochastic process is a key characteristic of its dynamics, of which entropy rate is a special fractal dimension named information dimension. The paper introduces methods of statistical physics to compute the multi-fractal of stochastic process so that the entropy rate of source can be obtained at once. Take binary hidden Markov processes as example, the paper demonstrate how this approach works. The results shows that the methods is applicable to numerically approximate the entropy rate of binary hidden Markov processes (BHMPs) in practical applications, and it can be applied in more generalized kinds of information sources.
An Entropy-Based Approach for Nonparametrically Testing Simple Probability Distribution Hypotheses

Ron Mittelhammer,George Judge,Miguel Henry

DOI: https://doi.org/10.48550/arXiv.2201.06647

2022-01-17

Econometrics

Abstract:In this paper, we introduce a flexible and widely applicable nonparametric entropy-based testing procedure that can be used to assess the validity of simple hypotheses about a specific parametric population distribution. The testing methodology relies on the characteristic function of the population probability distribution being tested and is attractive in that, regardless of the null hypothesis being tested, it provides a unified framework for conducting such tests. The testing procedure is also computationally tractable and relatively straightforward to implement. In contrast to some alternative test statistics, the proposed entropy test is free from user-specified kernel and bandwidth choices, idiosyncratic and complex regularity conditions, and/or choices of evaluation grids. Several simulation exercises were performed to document the empirical performance of our proposed test, including a regression example that is illustrative of how, in some contexts, the approach can be applied to composite hypothesis-testing situations via data transformations. Overall, the testing procedure exhibits notable promise, exhibiting appreciable increasing power as sample size increases for a number of alternative distributions when contrasted with hypothesized null distributions. Possible general extensions of the approach to composite hypothesis-testing contexts, and directions for future work are also discussed.
An interval method to measure the uncertainty of basic probability assignment

Jinyan Su,Yong Deng

DOI: https://doi.org/10.1007/s00500-022-07114-8

IF: 3.732

2022-01-01

Soft Computing

Abstract:Comparing the probability distribution, basic probability assignment in evidence theory is more efficient to deal with uncertain information. However, the uncertainty measure of basic probability assignment is still an open issue. In this paper, a new uncertainty measure based on Tsallis entropy is proposed to solve problems when the basic probability assignments are not given or being transformed into interval probability distribution. The proposed entropy is an extension of Tsallis entropy in continuous space. Some numerical examples are illustrated to show the efficiency of the proposed method.
Relative entropy of continuous distributions under knightian uncertainty

Gui Ping Tao,Liyan Han

2010-01-01

Abstract:Knightian uncertainty means a decision maker cannot choose a single probability distribution from a set of subjective probability distributions for measuring the natural states of the economy. The relative entropy is a measure to quantify how close a probability distribution is to a candidate distribution. Considering it cannot be calculated easily for many cases, it is of great value to discuss its approximation. An important property of the relative entropy, that is invariable under a strict monotone and differential transform of random variable, was proved. Then, the simplified calculation and approximation of the relative entropy were given by the Taylor formula. Finally, the neighborhood of probability distributions was described by relative entropy, and a homeomorphic map from an abstract probability measure space to its concrete parameters space was established.
Entropy Estimate for Degenerate SDEs with Applications to Nonlinear Kinetic Fokker–Planck Equations

Zhongmin Qian,Panpan Ren,Feng-Yu Wang

DOI: https://doi.org/10.1137/24m1634473

IF: 2.071

2024-07-20

SIAM Journal on Mathematical Analysis

Abstract:SIAM Journal on Mathematical Analysis, Volume 56, Issue 4, Page 5330-5349, August 2024. The relative entropy for two different degenerate diffusion processes is estimated by using the Wasserstein distance of initial distributions and the difference between coefficients. As applications, the entropy-cost inequality and exponential ergodicity in entropy are derived for distribution dependent stochastic Hamiltonian systems associated with nonlinear kinetic Fokker–Planck equations.

mathematics, applied
Normal Distribution Based on Maximum Deng Entropy

Siran Li,Fuyuan Xiao

DOI: https://doi.org/10.1016/j.chaos.2022.113057

IF: 9.922

2023-01-01

Chaos Solitons & Fractals

Abstract:We define the distribution of uncertainty based on Deng entropy and proved that the limiting form of the distribution is the normal distribution by the characteristic function. We conduct some discussion on the distribution, and we further analyze the error of its normal approximation and give the conditions that should be satisfied for the requirement of given error. Finally, we reveal the relationship between the distribution and the binomial distribution and the advantages of taking it as a prior distribution in evidence theory.
What is Entropy? A new perspective from games of chance

Sarah Brandsen,Isabelle Jianing Geng,Gilad Gour

DOI: https://doi.org/10.1103/PhysRevE.105.024117

2021-03-16

Abstract:Given entropy's central role in multiple areas of physics and science, one important task is to develop a systematic and unifying approach to defining entropy. Games of chance become a natural candidate for characterising the uncertainty of a physical system, as a system's performance in gambling games depends solely on the uncertainty of its output. In this work, we construct families of games which induce pre-orders corresponding to majorization, conditional majorization, and channel majorization. Finally, we provide operational interpretations for all pre-orders, show the relevance of these results to dynamical resource theories, and find the only asymptotically continuous classical dynamic entropy.

Quantum Physics,Mathematical Physics
A Note on Mathai's Entropy Measure

Hans J. Haubold

2024-10-25

Abstract:In a paper [8] the authors classify entropy into three categories, as a thermodynamics quantity, as a measure of information production, and as a means of statistical inference. An entropy measure introduced by Mathai falls into the second and third categories. It is shown that this entropy measure is the same whether the variables involved are real or complex scalar, vector, or matrix variables. If the entropy measure is optimized under some moment-like conditions then one can obtain various types of densities which are applicable in different areas. Unlike Tsallis' entropy [9], it does not need an intermediary escort distribution to yield the desired results. Calculus of variation can be directly applied to obtain the desired results under Mathai's entropy. Tsallis' entropy, which is the basis of the area of non-extensive statistical mechanics, is a modified version of the ?-generalized entropy of Havrda-Charvat considered in [7]. Various types of distributions that can be obtained through optimization of Mathai's entropy, are illustrated in this paper.

Statistical Mechanics,Data Analysis, Statistics and Probability
Entropy estimators for Markovian sequences: A comparative analysis

Juan De Gregorio,David Sanchez,Raul Toral

DOI: https://doi.org/10.3390/e26010079

2024-01-18

Abstract:Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy estimators are designed for sequences of independent events and their performance vary depending on the system being studied and the available data size. In this work we compare different entropy estimators and their performance when applied to Markovian sequences. Specifically, we analyze both binary Markovian sequences and Markovian systems in the undersampled regime. We calculate the bias, standard deviation and mean squared error for some of the most widely employed estimators. We discuss the limitations of entropy estimation as a function of the transition probabilities of the Markov processes and the sample size. Overall, this paper provides a comprehensive comparison of entropy estimators and their performance in estimating entropy for systems with memory, which can be useful for researchers and practitioners in various fields.

Statistical Mechanics,Chaotic Dynamics,Data Analysis, Statistics and Probability
Entropy Estimation via Uniformization

Ziqiao Ao,Jinglai Li

DOI: https://doi.org/10.1016/j.artint.2023.103954

2023-04-19

Abstract:Entropy estimation is of practical importance in information theory and statistical science. Many existing entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for high-dimensional problems. In this work we propose a transform-based method for high-dimensional entropy estimation, which consists of the following two main ingredients. First by modifying the k-NN based entropy estimator, we propose a new estimator which enjoys small estimation bias for samples that are close to a uniform distribution. Second we design a normalizing flow based mapping that pushes samples toward a uniform distribution, and the relation between the entropy of the original samples and the transformed ones is also derived. As a result the entropy of a given set of samples is estimated by first transforming them toward a uniform distribution and then applying the proposed estimator to the transformed samples. The performance of the proposed method is compared against several existing entropy estimators, with both mathematical examples and real-world applications.

Information Theory

Entropy and Probability Distributions

Properties of the Shannon, Rényi and other entropies: dependence in parameters, robustness in distributions and extremes

Entropy of function of uncertain variables.

Information-theoretic Methods for Estimating Complicated Probability Distributions

Sine Entropy for Uncertain Variables

Insights into Entropy As a Measure of Multivariate Variability.

Stochastic comparisons, differential entropy and varentropy for distributions induced by probability density functions

Entropy of Credibility Distributions for Fuzzy Variables

Entropy methods in random motion

Entropy generating function for past lifetime and its properties

Exponential entropy of uncertain sets and its applications to learning curve and portfolio optimization

Computing the Entropy Rate of Information Source with Methods of Statistical Physics

An Entropy-Based Approach for Nonparametrically Testing Simple Probability Distribution Hypotheses

An interval method to measure the uncertainty of basic probability assignment

Relative entropy of continuous distributions under knightian uncertainty

Entropy Estimate for Degenerate SDEs with Applications to Nonlinear Kinetic Fokker–Planck Equations

Normal Distribution Based on Maximum Deng Entropy

What is Entropy? A new perspective from games of chance

A Note on Mathai's Entropy Measure

Entropy estimators for Markovian sequences: A comparative analysis

Entropy Estimation via Uniformization