Information Entropy Algorithms for Image, Video, and Signal Processing

Gwanggil Jeon

DOI: https://doi.org/10.3390/e23080926

2021-07-21

Abstract:Information entropy is a basic concept in information theory associated with any random variable [...].

What problem does this paper attempt to address?

Generalized information entropy and generalized information dimension

Tianxiang Zhan,Jiefeng Zhou,Zhen Li,Yong Deng

DOI: https://doi.org/10.1016/j.chaos.2024.114976

2024-05-15

Abstract:The concept of entropy has played a significant role in thermodynamics and information theory, and is also a current research hotspot. Information entropy, as a measure of information, has many different forms, such as Shannon entropy and Deng entropy, but there is no unified interpretation of information from a measurement perspective. To address this issue, this article proposes Generalized Information Entropy (GIE) that unifies entropies based on mass function. Meanwhile, GIE establishes the relationship between entropy, fractal dimension, and number of events. Therefore, Generalized Information Dimension (GID) has been proposed, which extends the definition of information dimension from probability to mass fusion. GIE plays a role in approximation calculation and coding systems. In the application of coding, information from the perspective of GIE exhibits a certain degree of particle nature that the same event can have different representational states, similar to the number of microscopic states in Boltzmann entropy.

physics, mathematical, multidisciplinary,mathematics, interdisciplinary applications
Understanding Shannon's Entropy metric for Information

Sriram Vajapeyam

DOI: https://doi.org/10.48550/arXiv.1405.2061

2014-03-24

Information Theory

Abstract:Shannon's metric of "Entropy" of information is a foundational concept of information theory. This article is a primer for novices that presents an intuitive way of understanding, remembering, and/or reconstructing Shannon's Entropy metric for information.
Information Entropy in Chemistry: An Overview

Denis Sh. Sabirov,Igor S. Shepelevich

DOI: https://doi.org/10.3390/e23101240

IF: 2.738

2021-09-23

Entropy

Abstract:Basic applications of the information entropy concept to chemical objects are reviewed. These applications deal with quantifying chemical and electronic structures of molecules, signal processing, structural studies on crystals, and molecular ensembles. Recent advances in the mentioned areas make information entropy a central concept in interdisciplinary studies on digitalizing chemical reactions, chemico-information synthesis, crystal engineering, as well as digitally rethinking basic notions of structural chemistry in terms of informatics.

physics, multidisciplinary
Reflections on Shannon Information: In search of a natural information-entropy for images

Kieran G. Larkin

DOI: https://doi.org/10.48550/arXiv.1609.01117

2016-09-05

Abstract:It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete, bandlimited signal. Using Shannon's later theory of sampling we derive a new and symmetric version of the second order entropy in 1D. The new theory then naturally extends to 2D and higher dimensions, where by naturally we mean simple, symmetric, isotropic and parsimonious. Simplicity arises from the direct application of Shannon's joint entropy equalities and inequalities to the gradient (del) vector field image embodying the second order relations of the scalar image. Parsimony is guaranteed by halving of the vector data rate using Papoulis' generalized sampling expansion. The new 2D entropy measure, which we dub delentropy, is underpinned by a computable probability density function we call deldensity. The deldensity captures the underlying spatial image structure and pixel co-occurrence. It achieves this because each scalar image pixel value is nonlocally related to the entire gradient vector field. Both deldensity and delentropy are highly tractable and yield many interesting connections and useful inequalities. The new measure explicitly defines a realizable encoding algorithm and a corresponding reconstruction. Initial tests show that delentropy compares favourably with the conventional intensity-based histogram entropy and the compressed data rates of lossless image encoders (GIF, PNG, WEBP, JP2K-LS and JPG-LS) for a selection of images. The symmetric approach may have applications to higher dimensions and problems concerning image complexity measures.

Information Theory,Computer Vision and Pattern Recognition
Darboux transformation for the modified Veselov-Novikov equation

Delong Yu,Q. P. Liu,Shikun Wang

DOI: https://doi.org/10.1088/0305-4470/35/16/316

2001-11-08

Abstract:A Darboux transformation is constructed for the modified Veselov-Novikov equation.

Exactly Solvable and Integrable Systems
Lectures on Entropy. Part I

Vojkan Jaksic

DOI: https://doi.org/10.48550/arXiv.1806.07249

2018-06-16

Mathematical Physics

Abstract:These lecture notes concern information-theoretic notions of entropy. They are intended for, and have been successfully taught to, undergraduate students interested inresearch careers. Besides basic notions of analysis related to convergence that are typically taught in the first or second year of undergraduate studies, no other background is needed to read the notes. The notes might be also of interest to any mathematically inclined reader who wishes to learn basic facts about notions of entropy in an elementary setting.
An Information Theoretic Interpretation for H-Infinity Entropy

H Zhang,YX Sun

DOI: https://doi.org/10.1109/cacsd.2002.1036971

2002-01-01

Abstract:The relation between mutual information rate of system variables and system H-infinity entropy is formulated, an information theoretic interpretation of minimum entropy H-infinity control method is discussed.
Theory and Application of the Information Bottleneck Method

Jan Lewandowsky,Gerhard Bauch

DOI: https://doi.org/10.3390/e26030187

IF: 2.738

2024-02-23

Entropy

Abstract:Entropy, Vol. 26, Pages 187: Theory and Application of the Information Bottleneck Method Entropy doi: 10.3390/e26030187 Authors: Jan Lewandowsky Gerhard Bauch In 1999, Naftali Tishby et al [...]

physics, multidisciplinary
On the Structure of Information

Sebastian Gottwald,Daniel A. Braun

2024-09-30

Abstract:Shannon information and Shannon entropy are undoubtedly the most commonly used quantitative measures of information, cropping up in the literature across a broad variety of disciplines, often in contexts unrelated to coding theory. Here, we generalize the original idea behind Shannon entropy as the cost of encoding a sample of a random variable in terms of the required codeword length, to arbitrary loss functions by considering the optimally achievable loss given a certain level of knowledge about the random variable. By formalizing knowledge in terms of the measure-theoretic notion of sub-$\sigma$-algebras, we arrive at a general notion of uncertainty reduction that includes entropy and information as special cases: entropy is the reduction of uncertainty from no (or partial) knowledge to full knowledge about a random variable, whereas information is uncertainty reduction from no (or partial) knowledge to partial knowledge. As examples, we get Shannon information and entropy when measuring loss in terms of message length, variance for square error loss, and more generally, for the Bregman loss, we get Bregman information. Moreover, we show that appealing properties of Shannon entropy and information extend to the general case, including well-known relations involving the KL divergence, which are extended to divergences of proper scoring rules.

Information Theory
Relations between entropy rate, entropy production and information geometry in linear stochastic systems

Adrian-Josue Guel-Cortez,Eun-jin Kim

DOI: https://doi.org/10.1088/1742-5468/acbc24

2023-02-07

Abstract:In this work, we investigate the relation between the concept of ``information rate'', an information geometric method for measuring the speed of the time evolution of the statistical states of a stochastic process, and stochastic thermodynamics quantities like entropy rate and entropy production. Then, we propose the application of entropy rate and entropy production to different practical applications such as abrupt event detection, correlation analysis, and control engineering. Specifically, by utilising the Fokker-Planck equation of multi-variable linear stochastic processes described by Langevin equations, we calculate the exact value for information rate, entropy rate, and entropy production and derive various inequalities among them. Inspired by classical correlation coefficients and control techniques, we create entropic-informed correlation coefficients as abrupt event detection methods and information geometric cost functions as optimal thermodynamic control policies, respectively. The methods are analysed via the numerical simulations of common prototypical systems.

Statistical Mechanics
Information Geometry for the Working Information Theorist

Kumar Vijay Mishra,M. Ashok Kumar,Ting-Kam Leonard Wong

2023-10-06

Abstract:Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas such as radar sensing, array signal processing, quantum physics, deep learning, and optimal transport. This article presents an overview of essential information geometry to initiate an information theorist, who may be unfamiliar with this exciting area of research. We explain the concepts of divergences on statistical manifolds, generalized notions of distances, orthogonality, and geodesics, thereby paving the way for concrete applications and novel theoretical investigations. We also highlight some recent information-geometric developments, which are of interest to the broader information theory community.

Information Theory,Machine Learning,Signal Processing,Differential Geometry
Epsilon-Entropy and H∞ Entropy in Continuous Time Systems

Hui ZHANG,Youxian SUN

DOI: https://doi.org/10.3182/20080706-5-kr-1001.01354

2008-01-01

IFAC Proceedings Volumes

Abstract:Based on the analysis of ε – entropy, information in continuous time linear multivariable stochastic systems is discussed. To describe time average information variation after a process transmitted through a continuous time system, the concept of system variety defined by the difference between ε – entropy rates of system input and output is proposed. Furthermore, an equivalent relation between system variety and the H∞ entropy, an auxiliary performance index in H∞ control theory, is derived by using spectral factorization. This connection gives information theoretic interpretation to H∞ entropy, and provides new potential method to analyze and design continuous time stochastic systems.
Variety of linear continuous time systems

Hui Zhang,Youxian Sun

DOI: https://doi.org/10.1109/CDC.2009.5399639

2009-01-01

Abstract:The epsilon-entropy was adopted as a measure of information in the analysis of linear Gaussian continuous time control systems under the consideration that the `virtual reproductions' of system input and output are subjected to a common high enough precision requirement. The function of system variety which describes the variation of time average information in system was defined as the difference between epsilon-entropy rates of system input and output. It formulates how does the information or uncertainty of input transmit through the dynamic system and impact output. Relations within variety, Bode sensitivity integral and HÂ¿ entropy were then derived.
The Computational Theory of Intelligence: Information Entropy

Daniel Kovach

DOI: https://doi.org/10.4236/ijmnta.2014.34020

2014-12-24

Abstract:This paper presents an information theoretic approach to the concept of intelligence in the computational sense. We introduce a probabilistic framework from which computational intelligence is shown to be an entropy minimizing process at the local level. Using this new scheme, we develop a simple data driven clustering example and discuss its applications.

Artificial Intelligence,Machine Learning
Information Theoretic Learning: Reny's Entropy and Kernel Perspectives (principe, J.; 2010) [book Review]

Huajin Tang,Haizhou Li

DOI: https://doi.org/10.1109/mci.2011.941592

IF: 9.809

2011-01-01

IEEE Computational Intelligence Magazine

Abstract:This book, derived from Jose Principe and his group??s 10 years?? research in information theory and statistical learning, gives a comprehensive introduction, analysis and demonstration of almost all the major components required for understanding and developing the new theme of information-theoretical learning. The basic strategy utilized by the author is to apply information theory descriptors (namely entropy and divergence, in contrast to the statistical measures of mean and covariance)as nonparametric cost functions for the design of adaptive systems, thus creating a new paradigm of information theoretic learning. And like in statistical learning, unsupervised or supervised training modes are also fully explored.
Information entropy in cosmological inference problems

Ana Marta Pinho,Robert Reischke,Marie Teich,Björn Malte Schäfer

DOI: https://doi.org/10.48550/arXiv.2005.02035

2020-05-05

Abstract:The subject of this paper is a quantification of the information content of cosmological probes of the large-scale structures, specifically of temperature and polarisation anisotropies in the cosmic microwave background, CMB-lensing, weak cosmic shear and galaxy clustering, in terms of Information theory measures like information entropies. We aim to establish relationships for Gaussian likelihoods, between conventional measures of statistical uncertainties and information entropies. Furthermore, we extend these studies to the computation of (Bayesian) evidences and the power of measurement to distinguish between competing models. We investigate in detail how cosmological data decreases information entropy by reducing statistical errors and by breaking degeneracies. In addition, we work out how tensions between data sets increase information entropy and quantify this effect in three examples: the discrepancy in $\Omega_m$ and $\sigma_8$ between the CMB and weak lensing, the role of intrinsic alignments in weak lensing data when attempting the dark energy equation of state parameters, and the famous $H_0$-tension between Cepheids in the Hubble keystone project and the cosmic microwave background as observed by Planck.

Cosmology and Nongalactic Astrophysics
A mini-introduction to information theory

Edward Witten

DOI: https://doi.org/10.1007/s40766-020-00004-5

2020-03-23

La Rivista del Nuovo Cimento

Abstract:This article consists of a very short introduction to classical and quantum information theory. Basic properties of the classical Shannon entropy and the quantum von Neumann entropy are described, along with related concepts such as classical and quantum relative entropy, conditional entropy, and mutual information. A few more detailed topics are considered in the quantum case.
Weighted entropy: basic inequalities

Mark Kelbert,Izabella Stuhl,Yuri Suhov

DOI: https://doi.org/10.48550/arXiv.1710.10798

2017-10-30

Probability

Abstract:This paper represents an extended version of an earlier note [10]. The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. We analyse analogs of the Fisher information inequality and entropy power inequality for the weighted entropy and discuss connections with weighted Lieb's splitting inequality. The concepts of rates of the weighted entropy and information are also discussed.
Stochastic Information Gradient Algorithms Under (H, Phi)-Entropy Criterion

Badong Chen,Jinchun Hu,Hongbo Li,Zengqi Sun

2006-01-01

Abstract:The error entropy minimization approaches in adaptive systems have been extensively studied and widely used recently. In this paper, we introduced a general error entropy criterion, i.e. the (h, phi)-entropy criterion. The nonparametric estimator for (h, phi)-entropy with Parzen windowing and the associated stochastic information gradient (SIG) algorithm were presented, also the approximate upper bound for the step size in adaptive linear neuron (ADALINE) has been derived. Finally, we illustrated the advantages of this general SIG algorithm with certain h and phi functions in the prediction of time series and nonlinear system identification experiments.
Divergence, Entropy, Information: An Opinionated Introduction to Information Theory

Philip Chodrow

DOI: https://doi.org/10.48550/arXiv.1708.07459

2019-03-24

Abstract:Information theory is a mathematical theory of learning with deep connections with topics as diverse as artificial intelligence, statistical physics, and biological evolution. Many primers on information theory paint a broad picture with relatively little mathematical sophistication, while many others develop specific application areas in detail. In contrast, these informal notes aim to outline some elements of the information-theoretic "way of thinking," by cutting a rapid and interesting path through some of the theory's foundational concepts and results. They are aimed at practicing systems scientists who are interested in exploring potential connections between information theory and their own fields. The main mathematical prerequisite for the notes is comfort with elementary probability, including sample spaces, conditioning, and expectations. We take the Kullback-Leibler divergence as our most basic concept, and then proceed to develop the entropy and mutual information. We discuss some of the main results, including the Chernoff bounds as a characterization of the divergence; Gibbs' Theorem; and the Data Processing Inequality. A recurring theme is that the definitions of information theory support natural theorems that sound ``obvious'' when translated into English. More pithily, ``information theory makes common sense precise.'' Since the focus of the notes is not primarily on technical details, proofs are provided only where the relevant techniques are illustrative of broader themes. Otherwise, proofs and intriguing tangents are referenced in liberally-sprinkled footnotes. The notes close with a highly nonexhaustive list of references to resources and other perspectives on the field.

Information Theory,Statistics Theory,Data Analysis, Statistics and Probability

Information Entropy Algorithms for Image, Video, and Signal Processing

Generalized information entropy and generalized information dimension

Understanding Shannon's Entropy metric for Information

Information Entropy in Chemistry: An Overview

Reflections on Shannon Information: In search of a natural information-entropy for images

Darboux transformation for the modified Veselov-Novikov equation

Lectures on Entropy. Part I

An Information Theoretic Interpretation for H-Infinity Entropy

Theory and Application of the Information Bottleneck Method

On the Structure of Information

Relations between entropy rate, entropy production and information geometry in linear stochastic systems

Information Geometry for the Working Information Theorist

Epsilon-Entropy and H∞ Entropy in Continuous Time Systems

Variety of linear continuous time systems

The Computational Theory of Intelligence: Information Entropy

Information Theoretic Learning: Reny's Entropy and Kernel Perspectives (principe, J.; 2010) [book Review]

Information entropy in cosmological inference problems

A mini-introduction to information theory

Weighted entropy: basic inequalities

Stochastic Information Gradient Algorithms Under (H, Phi)-Entropy Criterion

Divergence, Entropy, Information: An Opinionated Introduction to Information Theory