Power Law and Entropy

Sachio Hirokawa,Eisuke Ito

DOI: https://doi.org/10.48550/arXiv.1504.06543

2015-04-22

Abstract:Shannon(1951) and Yavuz(1974) estimated the entropy of real documents. This note drives an upper bound of entropy from the parameter of the power law.

Information Theory

What problem does this paper attempt to address?

Entropy-power inequality for weighted entropy

Yuri Suhov,Salimeh Yasaei Sekeh,Mark Kelbert

DOI: https://doi.org/10.48550/arXiv.1502.02188

2015-02-07

Information Theory

Abstract:We analyse an analog of the entropy-power inequality for the weighted entropy.
A generalization of Costa's Entropy Power Inequality

Luca Tamanini

DOI: https://doi.org/10.48550/arXiv.2012.12230

2020-12-22

Information Theory

Abstract:Aim of this short note is to study Shannon's entropy power along entropic interpolations, thus generalizing Costa's concavity theorem. We shall provide two proofs of independent interest: the former by $\Gamma$-calculus, hence applicable to more abstract frameworks; the latter with an explicit remainder term, reminiscent of [20], allowing us to characterize the case of equality.
Weighted $p$-Rényi Entropy Power Inequality: Information Theory to Quantum Shannon Theory

Junseo Lee,Hyeonjun Yeo,Kabgyun Jeong

DOI: https://doi.org/10.1007/s10773-023-05512-8

2023-11-11

Abstract:We study the $p$-Rényi entropy power inequality with a weight factor $t$ on two independent continuous random variables $X$ and $Y$. The extension essentially relies on a modulation on the sharp Young's inequality due to Bobkov and Marsiglietti. Our research provides a key result that can be used as a fundamental research finding in quantum Shannon theory, as it offers a Rényi version of the entropy power inequality for quantum systems.

Quantum Physics
Properties of the Shannon, Rényi and other entropies: dependence in parameters, robustness in distributions and extremes

Iryna Bodnarchuk,Yuliya Mishura,Kostiantyn Ralchenko

2024-11-24

Abstract:We calculate and analyze various entropy measures and their properties for selected probability distributions. The entropies considered include Shannon, Rényi, generalized Rényi, Tsallis, Sharma-Mittal, and modified Shannon entropy, along with the Kullback-Leibler divergence. These measures are examined for several distributions, including gamma, chi-squared, exponential, Laplace, and log-normal distributions. We investigate the dependence of the entropy on the parameters of the respective distribution. We also study the convergence of Shannon entropy for certain probability distributions. Furthermore, we identify the extreme values of Shannon entropy for Gaussian vectors.

Information Theory,Probability
A Conditional Entropy Power Inequality for Dependent Variables

Oliver Johnson,O. Johnson

DOI: https://doi.org/10.1109/tit.2004.831790

IF: 2.5

2004-08-01

IEEE Transactions on Information Theory

Abstract:We provide a condition under which a version of Shannon's entropy power inequality will hold for dependent variables. We first provide a Fisher information inequality extending that found in the independent case. The key ingredients are a conditional expectation representation for the score function of a sum, and the de Bruijn identity which relates entropy and Fisher information.

computer science, information systems,engineering, electrical & electronic
Logarithmic and power-law entropies from convexity

Nikolaos Kalogeropoulos

2024-06-25

Abstract:In an attempt to understand the origin and robustness of the Boltzmann/Gibbs/Shannon entropic functional, we adopt a geometric approach and discuss the implications of the Johnson-Lindenstrauss lemma and of Dvoretzky's theorem on convex bodies for the choice of this functional form. We contrast these results with a more recent result on flowers of balls, which may be interpreted as suggesting the use of power-law entropies for some systems.

Statistical Mechanics,High Energy Physics - Theory,Mathematical Physics,Probability,History and Philosophy of Physics
Lectures on Entropy. Part I

Vojkan Jaksic

DOI: https://doi.org/10.48550/arXiv.1806.07249

2018-06-16

Mathematical Physics

Abstract:These lecture notes concern information-theoretic notions of entropy. They are intended for, and have been successfully taught to, undergraduate students interested inresearch careers. Besides basic notions of analysis related to convergence that are typically taught in the first or second year of undergraduate studies, no other background is needed to read the notes. The notes might be also of interest to any mathematically inclined reader who wishes to learn basic facts about notions of entropy in an elementary setting.
Power-law Energy Level Spacing Distributions in Fractals

Askar A. Iliasov,Mikhail I. Katsnelson,Shengjun Yuan

DOI: https://doi.org/10.1103/physrevb.99.075402

2019-01-01

Abstract:In this article we investigate the energy spectrum statistics of fractals at the quantum level. We show that the energy-level distribution of a fractal follows a power-law behaviour, if its energy spectrum is a limit set of piece-wise linear functions. We propose that such a behaviour is a general feature of fractals, which can not be described properly by random matrix theory. Several other arguments for the power-law behaviour of the energy level-spacing distributions are proposed.
Weighted entropy: basic inequalities

Mark Kelbert,Izabella Stuhl,Yuri Suhov

DOI: https://doi.org/10.48550/arXiv.1710.10798

2017-10-30

Probability

Abstract:This paper represents an extended version of an earlier note [10]. The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. We analyse analogs of the Fisher information inequality and entropy power inequality for the weighted entropy and discuss connections with weighted Lieb's splitting inequality. The concepts of rates of the weighted entropy and information are also discussed.
Power-Law Distributions in Empirical Data

Aaron Clauset,Cosma Rohilla Shalizi,M. E. J. Newman

DOI: https://doi.org/10.1137/070710111

IF: 10.2

2009-11-04

SIAM Review

Abstract:Power-law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man-made phenomena. Unfortunately, the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distributionthe part of the distribution representing large but rare eventsand by the difficulty of identifying the range over which power-law behavior holds. Commonly used methods for analyzing power-law data, such as least-squares fitting, can produce substantially inaccurate estimates of parameters for power-law distributions, and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all. Here we present a principled statistical framework for discerning and quantifying power-law behavior in empirical data. Our approach combines maximum-likelihood fitting methods with goodness-of-fit tests based on the KolmogorovSmirnov (KS) statistic and likelihood ratios. We evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches. We also apply the proposed methods to twenty-four real-world data sets from a range of different disciplines, each of which has been conjectured to follow a power-law distribution. In some cases we find these conjectures to be consistent with the data, while in others the power law is ruled out.

mathematics, applied
Conditional Law of Large Numbers and Entropy of Markov Processes

Jinwen Chen,Sen Dan

DOI: https://doi.org/10.1142/s0219025720500113

2020-01-01

Abstract:The importance of a law of large numbers is well known. In this paper, conditional law of large numbers for Markov processes is proved, which can be used in computing quantities related to sub-Markov sequences. A variational interpretation for this limit is given, which shows that typically the quantities of interests tend to minimize a certain entropy. A quasi-compact operator argument is involved.
Entropy Principle in Direct Derivation of Benford's Law

Oded Kafri

DOI: https://doi.org/10.48550/arXiv.0901.3047

2009-03-08

Abstract:The uneven distribution of digits in numerical data, known as Benford's law, was discovered in 1881. Since then, this law has been shown to be correct in copious numerical data relating to economics, physics and even prime numbers. Although it attracts considerable attention, there is no a priori probabilistic criterion when a data set should or should not obey the law. Here a general criterion is suggested, namely that any file of digits in the Shannon limit (namely, having maximum entropy) has a Benford's law distribution of digits.

Discrete Mathematics,Data Analysis, Statistics and Probability
Thinning, photonic beamsplitting, and a general discrete entropy power inequality

Saikat Guha,Jeffrey H. Shapiro,Raul Garcia-Patron Sanchez

DOI: https://doi.org/10.1109/ISIT.2016.7541390

2016-05-25

Abstract:Many partially-successful attempts have been made to find the most natural discrete-variable version of Shannon's entropy power inequality (EPI). We develop an axiomatic framework from which we deduce the natural form of a discrete-variable EPI and an associated entropic monotonicity in a discrete-variable central limit theorem. In this discrete EPI, the geometric distribution, which has the maximum entropy among all discrete distributions with a given mean, assumes a role analogous to the Gaussian distribution in Shannon's EPI. The entropy power of $X$ is defined as the mean of a geometric random variable with entropy $H(X)$. The crux of our construction is a discrete-variable version of Lieb's scaled addition $X \boxplus_\eta Y$ of two discrete random variables $X$ and $Y$ with $\eta \in (0, 1)$. We discuss the relationship of our discrete EPI with recent work of Yu and Johnson who developed an EPI for a restricted class of random variables that have ultra-log-concave (ULC) distributions. Even though we leave open the proof of the aforesaid natural form of the discrete EPI, we show that this discrete EPI holds true for variables with arbitrary discrete distributions when the entropy power is redefined as $e^{H(X)}$ in analogy with the continuous version. Finally, we show that our conjectured discrete EPI is a special case of the yet-unproven Entropy Photon-number Inequality (EPnI), which assumes a role analogous to Shannon's EPI in capacity proofs for Gaussian bosonic (quantum) channels.

Information Theory,Quantum Physics
Properties of Shannon and Rényi entropies of the Poisson distribution as the functions of intensity parameter

Volodymyr Braiman,Anatoliy Malyarenko,Yuliya Mishura,Yevheniia Anastasiia Rudyk

2024-02-06

Abstract:We consider two types of entropy, namely, Shannon and Rényi entropies of the Poisson distribution, and establish their properties as the functions of intensity parameter. More precisely, we prove that both entropies increase with intensity. While for Shannon entropy the proof is comparatively simple, for Rényi entropy, which depends on additional parameter $\alpha>0$, we can characterize it as nontrivial. The proof is based on application of Karamata's inequality to the terms of Poisson distribution.

Information Theory,Probability
The entropy power conjecture implies the McKean conjecture

Guillaume Wang

2024-08-26

Abstract:After reviewing the entropy power, the McKean, and the Gaussian completely monotone conjectures, we prove that the first implies the second, for each order of the time-derivative. The proof is elementary and is based on manipulating the Bell polynomials.

Information Theory
Entropy Jump and Entropic Central Limit Theorem for Independent Sum

Liuquan Yao,Shuai Yuan

2024-02-22

Abstract:It is a manuscript for results about entropic central limit theorem for independent sum under finite Poincaré constant conditions.

Probability,Information Theory
A Note on Mathai's Entropy Measure

Hans J. Haubold

2024-10-25

Abstract:In a paper [8] the authors classify entropy into three categories, as a thermodynamics quantity, as a measure of information production, and as a means of statistical inference. An entropy measure introduced by Mathai falls into the second and third categories. It is shown that this entropy measure is the same whether the variables involved are real or complex scalar, vector, or matrix variables. If the entropy measure is optimized under some moment-like conditions then one can obtain various types of densities which are applicable in different areas. Unlike Tsallis' entropy [9], it does not need an intermediary escort distribution to yield the desired results. Calculus of variation can be directly applied to obtain the desired results under Mathai's entropy. Tsallis' entropy, which is the basis of the area of non-extensive statistical mechanics, is a modified version of the ?-generalized entropy of Havrda-Charvat considered in [7]. Various types of distributions that can be obtained through optimization of Mathai's entropy, are illustrated in this paper.

Statistical Mechanics,Data Analysis, Statistics and Probability
Upper bound for entropy production in Markov processes

Tomohiro Nishiyama,Yoshihiko Hasegawa

2023-10-06

Abstract:The second law of thermodynamics states that entropy production cannot be negative. Recent developments concerning uncertainty relations in stochastic thermodynamics, such as thermodynamic uncertainty relations and speed limits, have yielded refined second laws that provide lower bounds of entropy production by incorporating information from current statistics or distributions. In contrast, in this study, we bound the entropy production from above by terms comprising the dynamical activity and maximum transition-rate ratio. We derive two upper bounds: one applies to steady-state conditions, whereas the other applies to arbitrary time-dependent conditions. We verify these bounds through numerical simulation and identify several potential applications.

Statistical Mechanics
Shannon Entropy Characteristics Of Two-Phase Flow Systems

Zuoyi Zhang,Lei Shi

DOI: https://doi.org/10.1063/1.370552

IF: 2.877

1999-01-01

Journal of Applied Physics

Abstract:A method to calculate Shannon entropy of two-phase flow systems from the power spectral density of the output signal is introduced in this article. Two series of experiments are made successively. One is used to investigate the density wave instability in a test facility to study the Chinese 200 MW nuclear heating reactor. After studying the Shannon entropy of the inlet pressure drop in all 534 experimental tests under different heating power, inlet subcooling, operating pressure, and inlet orifice resistance, we find that tests with higher negative Shannon entropy (negentropy) are unstable while those with lower negentropy are stable, like energy used in many fields. The other experiment was made to research two-phase flow regimes in vertical pipes. We calculate the time-dependent negentropy of the air-water flow through the power spectral density of the pressure drop. From the results, we find that the negentropy of the bubble flow is the smallest, the slug flow's negentropy is the largest, and the negentropy of the annular flow is between the bubble flow and annular flow. (C) 1999 American Institute of Physics. [S0021-8979(99)07411-3].
Entropy, von Neumann and the von Neumann entropy

D. Petz

DOI: https://doi.org/10.48550/arXiv.math-ph/0102013

2001-02-10

Abstract:This paper is an introduction to the von Neumann entropy in a historic approach. Von Neumann's gedanken experiment is repeated, which led him to the formula of thermodynamic entropy of a statistical operator. In the analysis of his ideas we stress the role of superselection sectors and summarize von Neumann's knowledge about quantum mechanical entropy. The final part of the paper is devoted to important developments discovered long after von Neumann's work. Subadditivity and the interpretation of the von Neumann entropy as channel capacity are among those.

Mathematical Physics

Power Law and Entropy

Entropy-power inequality for weighted entropy

A generalization of Costa's Entropy Power Inequality

Weighted $p$-Rényi Entropy Power Inequality: Information Theory to Quantum Shannon Theory

Properties of the Shannon, Rényi and other entropies: dependence in parameters, robustness in distributions and extremes

A Conditional Entropy Power Inequality for Dependent Variables

Logarithmic and power-law entropies from convexity

Lectures on Entropy. Part I

Power-law Energy Level Spacing Distributions in Fractals

Weighted entropy: basic inequalities

Power-Law Distributions in Empirical Data

Conditional Law of Large Numbers and Entropy of Markov Processes

Entropy Principle in Direct Derivation of Benford's Law

Thinning, photonic beamsplitting, and a general discrete entropy power inequality

Properties of Shannon and Rényi entropies of the Poisson distribution as the functions of intensity parameter

The entropy power conjecture implies the McKean conjecture

Entropy Jump and Entropic Central Limit Theorem for Independent Sum

A Note on Mathai's Entropy Measure

Upper bound for entropy production in Markov processes

Shannon Entropy Characteristics Of Two-Phase Flow Systems

Entropy, von Neumann and the von Neumann entropy