Information vs Thermodynamic Entropy

Phil Attard

2024-07-12

Abstract:The Shannon information is shown to be different to the thermodynamic entropy, and indifferent to the Second Law of Thermodynamics.

Statistical Mechanics,Data Analysis, Statistics and Probability,Quantum Physics

What problem does this paper attempt to address?

Information entropy and thermal entropy: apples and oranges

Laszlo B. Kish,David K. Ferry

DOI: https://doi.org/10.48550/arXiv.1706.01459

2017-07-18

Abstract:The frequent misunderstanding of information entropy is pointed out. It is shown that, contrary to fortuitous situations and common beliefs, there is no general interrelation between the information entropy and the thermodynamical entropy. We point out that the change of information entropy during measurement is determined by the resolution of the measurement instrument and these changes do not have a general and clear-cut interrelation with the change of thermal entropy. Moreover, these changes can be separated in space and time. We show classical physical considerations and examples to support this conclusion. Finally, we show that information entropy can violate the Third Law of Thermodynamics which is another indication of major differences from thermal entropy.

General Physics
Axiomatic Relation between Thermodynamic and Information-Theoretic Entropies

Mirjam Weilenmann,Lea Kraemer,Philippe Faist,Renato Renner

DOI: https://doi.org/10.1103/physrevlett.117.260601

IF: 8.6

2016-12-22

Physical Review Letters

Abstract:Thermodynamic entropy, as defined by Clausius, characterizes macroscopic observations of a system based on phenomenological quantities such as temperature and heat. In contrast, information-theoretic entropy, introduced by Shannon, is a measure of uncertainty. In this Letter, we connect these two notions of entropy, using an axiomatic framework for thermodynamics [E. H. Lieb and J. Yngvason Proc. R. Soc. 469, 20130408 (2013)]. In particular, we obtain a direct relation between the Clausius entropy and the Shannon entropy, or its generalization to quantum systems, the von Neumann entropy. More generally, we find that entropy measures relevant in nonequilibrium thermodynamics correspond to entropies used in one-shot information theory.

physics, multidisciplinary
Information form of the second law of thermodynamics

Miroslav Holeček

DOI: https://doi.org/10.48550/arXiv.2103.16913

2022-04-28

Abstract:An essential role of information in microscopic thermodynamics (e.g. Maxwell's demon) opens a challenging question if there exists a formulation of the second law of thermodynamics based only on pure information ideas. Here, such a formulation is suggested for unitary processes by introducing information as a full-valuable physical quantity defining an (objective) microscopic information entropy as 'information about microstate'. We show that various forms of entropy (Boltzmann, Shannon, Clausius) are in fact only a special cases of information entropy whose general form is found out. An observer plays here the role of a special (information) reference frame (IRF) towards which the entropy is defined. Some paradoxes or misunderstandings connected with the concept of entropy or the content of the second law arise by describing a situation without specifying a concrete IRF. Typically, the Boltzmann statistical approach cannot be symmetrically used towards the past as towards the future in one IRF. The information second law is full usable at meso- or microscopic scales: the information form of the generalized second law is found out too.

Statistical Mechanics
On the Supposed Mass of Entropy and That of Information

Didier Lairez

DOI: https://doi.org/10.3390/e26040337

IF: 2.738

2024-04-16

Entropy

Abstract:In the theory of special relativity, energy can be found in two forms: kinetic energy and rest mass. The potential energy of a body is actually stored in the form of rest mass, the interaction energy too, but temperature is not. Information acquired about a dynamical system can be potentially used to extract useful work from it. Hence, the "mass–energy–information equivalence principle" that has been recently proposed. In this paper, it is first recalled that for a thermodynamic system made of non-interacting entities at constant temperature, the internal energy is also constant. So, the energy involved in a variation in entropy (TΔS) differs from a change in the potential energy stored or released and cannot be associated to a corresponding variation in mass of the system, even if it is expressed in terms of the quantity of information. This debate gives us the opportunity to deepen the notion of entropy seen as a quantity of information, to highlight the difference between logical irreversibility (a state-dependent property) and thermodynamical irreversibility (a path-dependent property), and to return to the nature of the link between energy and information that is dynamical.

physics, multidisciplinary
On Thermodynamic Information

Bing Miao,Hong Qian,Yong-Shi Wu

2023-12-06

Abstract:Information based thermodynamic logic is revisited. It consists of two parts: Part A applies the modern theory of probability in which an arbitrary convex function \phi is employed as an analytic "device" to express information as statistical dependency contained in the topological sub-\sigma-algebra structure. Via thermo-doubling, Fenchel-Young equality (FYE) that consists of \phi(x) and its conjugate \psi(y) establishes the notion of equilibrium between x and y through duality symmetry and the principle of maximum entropy/minimum free energy. Part B deals with a given set of repetitive measurements, where an inherent convex function emerges via the mathematics of large deviations. Logarithm-based Shannon entropy with \phi(x)=-\log x figures prominently for i.i.d. sample statistics. Information can be a measure of the agreement between a statistical observation and its theoretical models. Maximum likelihood principle arises here and FYE provides a thermodynamic energetic narrative of recurrent data.

Statistical Mechanics,Mathematical Physics
Thermodynamics of Information

Juan M. R. Parrondo

2023-06-21

Abstract:As early as 1867, two years after the introduction of the concept of entropy by Clausius, Maxwell showed that the limitations imposed by the second law of thermodynamics depend on the information that one possesses about the state of a physical system. A "very observant and neat-fingered being", later on named Maxwell demon by Kelvin, could arrange the molecules of a gas and induce a temperature or pressure gradient without performing work, in apparent contradiction to the second law. One century later, Landauer claimed that "information is physical", and showed that certain processes involving information, like overwriting a memory, need work to be completed and are unavoidably accompanied by heat dissipation. Thermodynamics of information analyzes this bidirectional influence between thermodynamics and information processing. The seminal ideas that Landauer and Bennett devised in the 1970s have been recently reformulated in a more precise and general way by realizing that informational states are out of equilibrium and applying new tools from non-equilibrium statistical mechanics.

Statistical Mechanics,History and Philosophy of Physics
Second-law-like inequalities with information and their interpretations

Jordan M. Horowitz,Henrik Sandberg

DOI: https://doi.org/10.1088/1367-2630/16/12/125007

2014-09-18

Abstract:In a thermodynamic process with measurement and feedback, the second law of thermodynamics is no longer valid. In its place, various second-law-like inequalities have been advanced that each incorporate a distinct additional term accounting for the information gathered through measurement. We quantitatively compare a number of these information measures using an analytically tractable model for the feedback cooling of a Brownian particle. We find that the information measures form a hierarchy that reveals a web of interconnections. To untangle their relationships, we address the origins of the information, arguing that each information measure represents the minimum thermodynamic cost to acquire that information through a separate, distinct measurement protocol.

Statistical Mechanics
Understanding Shannon's Entropy metric for Information

Sriram Vajapeyam

DOI: https://doi.org/10.48550/arXiv.1405.2061

2014-03-24

Information Theory

Abstract:Shannon's metric of "Entropy" of information is a foundational concept of information theory. This article is a primer for novices that presents an intuitive way of understanding, remembering, and/or reconstructing Shannon's Entropy metric for information.
Variability as a better characterization of Shannon entropy

Gabriele Carcassi,Christine A Aidala,Julian Barbour

DOI: https://doi.org/10.1088/1361-6404/abe361

2021-05-19

European Journal of Physics

Abstract:Abstract The Shannon entropy, one of the cornerstones of information theory, is widely used in physics, particularly in statistical mechanics. Yet its characterization and connection to physics remain vague, leaving ample room for misconceptions and misunderstanding. We will show that the Shannon entropy can be fully understood as measuring the variability of the elements within a given distribution: it characterizes how much variation can be found within a collection of objects. We will see that it is the only indicator that is continuous and linear, that it quantifies the number of yes/no questions (i.e. bits) that are needed to identify an element within the distribution, and we will see how applying this concept to statistical mechanics in different ways leads to the Boltzmann, Gibbs and von Neumann entropies.

physics, multidisciplinary,education, scientific disciplines
The Exact Entropy Formula of the Ideal Gas and its Information-Theoretic Interpretation

Arnaldo Spalvieri

2023-05-17

Abstract:The paper analyzes the entropy of a system composed by non-interacting and indistinguishable particles whose quantum state numbers are modelled as independent and identically distributed classical random variables. The crucial observation is that, under this assumption, whichever is the number of particles that constitute the system, the occupancy numbers of system's quantum (micro)states are multinomially distributed. This observation leads to an entropy formula for the physical system, which is nothing else than the entropy formula of the multinomial distribution, for which we claim novelty, in the sense that it is proposed here for the first time that the entropy of the multinomial distribution is the entropy of the physical system. The entropy formula of the multinomial distribution unveils yet unexplored connections between information theory and statistical mechanics, among which we mention the connection between conditional entropy of the random microstate given the random occupancy numbers and the Boltzmann-Planck entropy $\log(W)$ and between these two and the Gibbs correction term $\log(N!)$, thermalization and communication-theoretic preparation of a thermal state, accessible information of the thermal state and physical entropy of the thermalized system. A noticeable specific result that descends from our approach is the exact quantum correction to the textbook Sackur-Tetrode formula for the entropy of an ideal gas at the thermal equilibrium in a container.

Statistical Mechanics
Interrelatedness of thermodynamics and information: transformation of heat as a measurable information process and quantity, an overview

B. Hejna,J. Šesták,Hejna, B.,Šesták, J.

DOI: https://doi.org/10.1007/s10973-023-12833-7

IF: 4.755

2024-03-05

Journal of Thermal Analysis and Calorimetry

Abstract:No one has yet investigated the information reach and influence of thermal analysis. Even so the heat and entropy are bound closely. The very problem of the term entropy is that it was defined specifically within the classical, equilibrium Thermodynamics like Clausius entropy, while its meaning is more general. It is usable in all areas of physics, in the communication theory and technology, and also, in social sciences, just as an universal quantitative model. The bridge to this wider world is in Boltzmann's statistical point of view used in the Shannon's information point of view. The difference is that the Shannon entropy is defined more generally, both for the uniform and for the not uniform probability distributions. But with renormalization and with the Boltzmann constant we can speak, using the Clausius entropy, about a certain thermodynamic system considered as a heat model of the area of our wider interests. So we will to deal with the heat and its entropy embodied to a wider consideration. In addition to the standard physical meaning, another thermodynamic realization in the form of a message is derived from Shannon's information theory, where the sample carries a measure of Shannon's information given by its structure or internal organization. This is also a contribution to improving the approach to thinking about the importance of observation and measurement, reflecting at the same time the costs and influence of the measurement itself and its organization on the measured object or, on its environment or, even on the environment of the measuring itself generally. Statistical sorting also carries with it the internal movement of individual elementary particles, the value of which indicates the resulting state of heat transfer and dissipation to and from the system providing a state of called equilibrium. It is statistically described by a weighted quantity called temperature, the changes of which via momentum alteration satisfy Newton's laws of inertia allowing enumeration of the related heat information processing. The terminology used is still non-institutional.

chemistry, physical, analytical,thermodynamics
Nonequilibrium in Thermodynamic Formalism: the Second Law, gases and Information Geometry

Artur O. Lopes,R. Ruggiero

DOI: https://doi.org/10.48550/arXiv.2103.08333

2021-11-24

Abstract:In Nonequilibrium Thermodynamics and Information Theory, the relative entropy (or, KL divergence) plays a very important role. Consider a Hölder Jacobian $J$ and the Ruelle (transfer) operator $\mathcal{L}_{\log J}.$ Two equilibrium probabilities $\mu_1$ and $\mu_2$, can interact via a discrete-time {\it Thermodynamic Operation} described by the action {\it of the dual of the Ruelle operator} $ \mathcal{L}_{\log J}^*$. We argue that the law $\mu \to \mathcal{L}_{\log J}^*(\mu)$, producing nonequilibrium, can be seen as a Thermodynamic Operation after showing that it's a manifestation of the Second Law of Thermodynamics. We also show that the change of relative entropy satisfies $$ D_{K L} (\mu_1,\mu_2) - D_{K L} (\mathcal{L}_{\log J}^*(\mu_1),\mathcal{L}_{\log J}^*(\mu_2))= 0.$$ Furthermore, we describe sufficient conditions on $J,\mu_1$ for getting $h(\mathcal{L}_{\log J}^*(\mu_1))\geq h(\mu_1)$, where $h$ is entropy. Recalling a natural Riemannian metric in the Banach manifold of Hölder equilibrium probabilities we exhibit the second-order Taylor formula for an infinitesimal tangent change of KL divergence; a crucial estimate in Information Geometry. We introduce concepts like heat, work, volume, pressure, and internal energy, which play here the role of the analogous ones in Thermodynamics of gases. We briefly describe the MaxEnt method.

Dynamical Systems,Statistical Mechanics,Information Theory,Mathematical Physics,Probability
Causation and Information Flow with Respect to Relative Entropy

X. San Liang

DOI: https://doi.org/10.1063/1.5010253

2018-01-01

Abstract:Recently, a rigorous formalism has been established for information flow and causality within dynamical systems with respect to Shannon entropy. In this study, we re-establish the formalism with respect to relative entropy, or Kullback-Leiber divergence, a well-accepted measure of predictability because of its appealing properties such as invariance upon nonlinear transformation and consistency with the second law of thermodynamics. Different from previous studies (which yield consistent results only for 2D systems), the resulting information flow, say T, is precisely the same as that with respect to Shannon entropy for systems of arbitrary dimensionality, except for a minus sign (reflecting the opposite notion of predictability vs. uncertainty). As before, T possesses a property called principle of nil causality, a fact that classical formalisms fail to verify in many situation. Besides, it proves to be invariant upon nonlinear transformation, indicating that the so-obtained information flow should be an intrinsic physical property. This formalism has been validated with the stochastic gradient system, a nonlinear system that admits an analytical equilibrium solution of the Boltzmann type.
Which entropy for general physical theories?

Paolo Perinotti,Alessandro Tosini,Leonardo Vaglini

2024-05-23

Abstract:We address the problem of quantifying the information content of a source for an arbitrary information theory, where the information content is defined in terms of the asymptotic achievable compression rate. The functions that solve this problem in classical and quantum theory are Shannon's and von Neumann's entropy, respectively. However, in a general information theory there are three different functions that extend the notion of entropy, and this opens the question as to whether any of them can universally play the role of the quantifier for the information content. Here we answer the question in the negative, by evaluating the information content as well as the various entropic functions in a toy theory called Bilocal Classical Theory.

Quantum Physics
Entanglement Theory and the Second Law of Thermodynamics

Fernando G.S.L. Brandao,Martin B. Plenio

DOI: https://doi.org/10.1038/nphys1100

2008-10-14

Abstract:Entanglement is central both to the foundations of quantum theory and, as a novel resource, to quantum information science. The theory of entanglement establishes basic laws, such as the non-increase of entanglement under local operations, that govern its manipulation and aims to draw from them formal analogies to the second law of thermodynamics. However, while in the second law the entropy uniquely determines whether a state is adiabatically accessible from another, the manipulation of entanglement under local operations exhibits a fundamental irreversibility which prevents the existence of such an order. Here we show that a reversible theory of entanglement and a rigorous relationship with thermodynamics may be established when one considers all non-entangling transformations. The role of the entropy in the second law is taken by the asymptotic relative entropy of entanglement in the basic law of entanglement. We show the usefulness of this new approach to general resource theories and to quantum information theory.

Quantum Physics
Entropy balance and Information processing in bipartite and non-bipartite composite systems

Richard E. Spinney,Joseph T. Lizier,Mikhail Prokopenko

DOI: https://doi.org/10.48550/arXiv.1712.09715

2017-12-28

Statistical Mechanics

Abstract:Information dynamics is an emerging description of information processing in complex systems which describes systems in terms of intrinsic computation, identifying computational primitives of information storage and transfer. In this paper we make a formal analogy between information dynamics and stochastic thermodynamics which describes the thermal behaviour of small irreversible systems. As stochastic dynamics is increasingly being utilized to quantify the thermodynamics associated with the processing of information we suggest such an analogy is instructive, highlighting that existing thermodynamic quantities can be described solely in terms of extant information theoretic measures related to information processing. In this contribution we construct irreversibility measures in terms of these quantities and relate them to the physical entropy productions that characterise the behaviour of single and composite systems in stochastic thermodynamics illustrating them with simple examples. Moreover, we can apply such a formalism to systems which do not have a bipartite structure. In particular we demonstrate that, given suitable non-bipartite processes, the heat flow in a subsystem can still be identified and one requires the present formalism to recover generalizations of the second law. In these systems residual irreversibility is associated with neither subsystem and this must be included in the these generalised second laws. This opens up the possibility of describing all physical systems in terms of computation allowing us to propose a framework for discussing the reversibility of systems traditionally out of scope of stochastic thermodynamics.
Signed Rényi Entropy and Quantum Second Laws

Adam Brandenburger,Pierfrancesco La Mura

2024-10-21

Abstract:We modify the Rényi (1961) axioms for entropy to apply to negative (``signed") measures as arise, for example, in phase-space representations of quantum mechanics. We obtain two new measures of (lack of) information about a system -- which we propose as signed analogs to classical Shannon entropy and classical Rényi entropy, respectively. We show that signed Rényi entropy witnesses non-classicality of a system. Specifically, a measure has at least one negative component if and only if signed Rényi $\alpha$-entropy is negative for some $\alpha > 1$. The corresponding non-classicality test does not work with signed Shannon entropy. We next show that signed Rényi $2k$-entropy, when $k$ is a positive integer, is Schur-concave. (An example shows that signed Shannon entropy is not Schur-concave.) We then establish an abstract quantum H-theorem for signed measures. We prove that signed Rényi $2k$-entropy is non-decreasing under classical (``decohering") evolution of a signed measure, where the latter could be a Wigner function or other phase-space representation of a quantum system. (An example shows that signed Shannon entropy may be non-monotonic.) We also provide a characterization of the Second Law for signed Rényi $2$-entropy in terms of what we call eventual classicalization of evolution of a system. We conclude with an argument that signed Rényi $2$-entropy of the Wigner function is constant under Moyal bracket evolution.

Quantum Physics,Mathematical Physics
Gibbs Paradox and the Concepts of Information, Symmetry, Similarity and Their Relationship

Shu-Kun Lin

DOI: https://doi.org/10.3390/entropy-e10010001

2008-03-18

Abstract:Information (I) is defined as the amount of the data after data compression. The first law of information theory: the total amount of data L (the sum of entropy S and information I) of an isolated system remains unchanged. The second law of information theory: Information I of an isolated system decreases to a minimum at equilibrium. The third law of information theory: For a solid structure of perfect symmetry (e.g., a perfect crystal), the information I is zero and the (information theory) entropy (called by me as static entropy for solid state) S is at the maximum. Gibbs Paradox has been resolved. Spontaneously mixed substances at gaseous state can be spontaneously separated at condensed phases (solid or liquid states), driving only by information loss or by the increase in (information theory) entropy. None of the typical pure mixing or separation processes are driving by free energy minimization and the free energy (or total amount of chemical potential) has no change during the processes of ideal mixture formation or ideal mixture separation. The thermodynamic entropy change for the formation of ideal mixtures of gases, liquids or solids is always zero.

Chemical Physics
Second Law of Thermodynamics without Einstein Relation

Benjamin Sorkin,Haim Diamant,Gil Ariel,Tomer Markovich

2024-07-28

Abstract:Materials that are constantly driven out of thermodynamic equilibrium, such as active and living systems, typically violate the Einstein relation. This may arise from active contributions to particle fluctuations which are unrelated to the dissipative resistance of the surrounding medium. We show that in these cases the widely used relation between informatic entropy production and heat dissipation does not hold. Consequently, fluctuation relations for the mechanical work, such as the Jarzynski and Crooks theorems, are invalid. We relate the breaking of the correspondence between entropy production and heat dissipation to departure from the fluctuation-dissipation theorem. We propose a temperature-like variable which restores this correspondence and gives rise to a generalized second law of thermodynamics, whereby the dissipated heat is necessarily nonnegative and vanishes at equilibrium. The Clausius inequality, Carnot maximum efficiency theorem, and relation between the extractable work and the change of free energy are recovered as well.

Statistical Mechanics,Soft Condensed Matter

Information vs Thermodynamic Entropy

Information entropy and thermal entropy: apples and oranges

Axiomatic Relation between Thermodynamic and Information-Theoretic Entropies

Information form of the second law of thermodynamics

On the Supposed Mass of Entropy and That of Information

On Thermodynamic Information

Thermodynamics of Information

Second-law-like inequalities with information and their interpretations

Understanding Shannon's Entropy metric for Information

Variability as a better characterization of Shannon entropy

The Exact Entropy Formula of the Ideal Gas and its Information-Theoretic Interpretation

Interrelatedness of thermodynamics and information: transformation of heat as a measurable information process and quantity, an overview

Nonequilibrium in Thermodynamic Formalism: the Second Law, gases and Information Geometry

Causation and Information Flow with Respect to Relative Entropy

Which entropy for general physical theories?

Entanglement Theory and the Second Law of Thermodynamics

Entropy balance and Information processing in bipartite and non-bipartite composite systems

Signed Rényi Entropy and Quantum Second Laws

Gibbs Paradox and the Concepts of Information, Symmetry, Similarity and Their Relationship

Second Law of Thermodynamics without Einstein Relation