Exponential weighted entropy and exponential weighted mutual information

YuShiwei,HuangTing-Zhu

IF: 6

2017-01-01

Neurocomputing

Abstract:In this paper, the exponential weighted entropy (EWE) and exponential weighted mutual information (EWMI) are proposed as the more generalized forms of Shannon entropy and mutual information (MI), r...

What problem does this paper attempt to address?

Exponential Weighted Entropy and Exponential Weighted Mutual Information.

Shiwei Yu,Ting-Zhu Huang

DOI: https://doi.org/10.1016/j.neucom.2017.03.075

IF: 6

2017-01-01

Neurocomputing

Abstract:In this paper, the exponential weighted entropy (EWE) and exponential weighted mutual information (EWMI) are proposed as the more generalized forms of Shannon entropy and mutual information (MI), respectively. They are position-related and causal systems that redefine the foundations of information-theoretic metrics. As the special forms of the weighted entropy and the weighted mutual information, EWE and EWMI have been proved that they preserve nonnegativity and concavity properties similar to Shannon frameworks. They can be adopted as the information measures in spatial interaction modeling. Paralleling with the normalized mutual information (NMI), the normalized exponential weighted mutual information (NEWMI) is also investigated. Image registration experiments demonstrate that EWMI and NEWMI algorithms can achieve higher aligned accuracy than MI and NMI algorithms.
Weighted entropy: basic inequalities

Mark Kelbert,Izabella Stuhl,Yuri Suhov

DOI: https://doi.org/10.48550/arXiv.1710.10798

2017-10-30

Probability

Abstract:This paper represents an extended version of an earlier note [10]. The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. We analyse analogs of the Fisher information inequality and entropy power inequality for the weighted entropy and discuss connections with weighted Lieb's splitting inequality. The concepts of rates of the weighted entropy and information are also discussed.
A Modified Belief Entropy in Dempster-Shafer Framework

Deyun Zhou,Yongchuan Tang,Wen Jiang

DOI: https://doi.org/10.1371/journal.pone.0176832

IF: 3.7

2017-01-01

PLoS ONE

Abstract:How to quantify the uncertain information in the framework of Dempster-Shafer evidence theory is still an open issue. Quite a few uncertainty measures have been proposed in Dempster-Shafer framework, however, the existing studies mainly focus on the mass function itself, the available information represented by the scale of the frame of discernment (FOD) in the body of evidence is ignored. Without taking full advantage of the information in the body of evidence, the existing methods are somehow not that efficient. In this paper, a modified belief entropy is proposed by considering the scale of FOD and the relative scale of a focal element with respect to FOD. Inspired by Deng entropy, the new belief entropy is consistent with Shannon entropy in the sense of probability consistency. What's more, with less information loss, the new measure can overcome the shortage of some other uncertainty measures. A few numerical examples and a case study are presented to show the efficiency and superiority of the proposed method.
Basic inequalities for weighted entropies

Yuri Suhov,Izabella Stuhl,Salimeh Yasaei Sekeh,Mark Kelbert

DOI: https://doi.org/10.1007/s00010-015-0396-5

2016-01-13

Aequationes Mathematicae

Abstract:The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. In this paper, we establish a number of simple inequalities for the weighted entropies (general as well as specific), mirroring similar bounds on standard (Shannon) entropies and related quantities. The required assumptions are written in terms of various expectations of weight functions. Examples are weighted Ky Fan and weighted Hadamard inequalities involving determinants of positive-definite matrices, and weighted Cramér-Rao inequalities involving the weighted Fisher information matrix.

mathematics, applied
Information Theoretic Proofs of Entropy Power Inequalities

Olivier Rioul

DOI: https://doi.org/10.1109/TIT.2010.2090193

2010-08-24

Abstract:While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verdú used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

Information Theory
Conditional Deng Entropy, Joint Deng Entropy and Generalized Mutual Information

Haoyang Zheng,Yong Deng

2016-01-01

Abstract:Shannon entropy, conditional entropy, joint entropy and mutual information, can estimate the chaotic level of information. However, these methods could only handle certain situations. Based on Deng entropy, this paper introduces multiple new entropy to estimate entropy under multiple interactive uncertain information: conditional Deng entropy is used to calculate entropy under conditional basic belief assignment; joint Deng entropy could calculate entropy by applying joint basic belief assignment distribution; generalized mutual information is applied to estimate the uncertainty of information under knowing another information. Numerical examples are used for illustrating the function of new entropy in the end.
Adaptive Exponential Weighted Mutual Information As a Similarity Measure for Medical Image Registration

陈芳,张建秋,胡波

DOI: https://doi.org/10.3969/j.issn.1674-1242.2008.03.003

2008-01-01

Progress in Biomedical Engineering

Abstract:In this paper,a new similarity measure for medical image registration,called Adaptive Expo- nential Weighted Mutual Information(AEWMI),is proposed.AEWMI could improve the smoothness and peak keenness performance of similarity measurement curve via adaptively weighting MI according to image quality and resolution.The experimental results,demonstrating the analyzing result,show that our method is robust to image noise or resolution difference and improves the registration robustness effectively.
Generalized information entropy and generalized information dimension

Tianxiang Zhan,Jiefeng Zhou,Zhen Li,Yong Deng

DOI: https://doi.org/10.1016/j.chaos.2024.114976

2024-05-15

Abstract:The concept of entropy has played a significant role in thermodynamics and information theory, and is also a current research hotspot. Information entropy, as a measure of information, has many different forms, such as Shannon entropy and Deng entropy, but there is no unified interpretation of information from a measurement perspective. To address this issue, this article proposes Generalized Information Entropy (GIE) that unifies entropies based on mass function. Meanwhile, GIE establishes the relationship between entropy, fractal dimension, and number of events. Therefore, Generalized Information Dimension (GID) has been proposed, which extends the definition of information dimension from probability to mass fusion. GIE plays a role in approximation calculation and coding systems. In the application of coding, information from the perspective of GIE exhibits a certain degree of particle nature that the same event can have different representational states, similar to the number of microscopic states in Boltzmann entropy.

physics, mathematical, multidisciplinary,mathematics, interdisciplinary applications
Shannon Shakes Hands with Chernoff: Big Data Viewpoint on Channel Information Measures.

Shanyun Liu,Rui She,Jiaxun Lu,Pingyi Fan

DOI: https://doi.org/10.48550/arxiv.1701.03237

2017-01-01

Abstract:Shannon entropy is the most crucial foundation of Information Theory, which has been proven to be effective in many fields such as communications. Renyi entropy and Chernoff information are other two popular measures of information with wide applications. The mutual information is effective to measure the channel information for the fact that it reflects the relation between output variables and input variables. In this paper, we reexamine these channel information measures in big data viewpoint by means of ACE algorithm. The simulated results show us that decomposition results of Shannon and Chernoff mutual information with respect to channel parametersare almost the same. In this sense, Shannon shakes hands with Chernoff since they are different measures of the same information quantity. We also propose a conjecture that there is nature of channel information which is only decided by the channel parameters.
Mutual Information Multinomial Estimation

Yanzhi Chen,Zijing Ou,Adrian Weller,Yingzhen Li

2024-08-18

Abstract:Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. This work proposes a new estimator for mutual information. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. This preliminary estimate serves as a bridge between the joint and the marginal distribution, and by comparing with this bridge distribution we can easily obtain the true difference between the joint distributions and the marginal distributions. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.

Machine Learning,Information Theory
Combination Weighting Approach Based on Deviation Function and Joint-Entropy

ZHANG Guoquan,LI Wenli,WANG Mingzheng

DOI: https://doi.org/10.3969/j.issn.1672-884X.2008.03.012

2008-01-01

Abstract:From assigning the weights of multiple-index correctly, a new method based on minimal deviations and maximal joint-entropy is proposed, in which the weight vectors at different point of views determined by several evaluation method was considered and the whole-warp between the ideal combination of the estimated weight vectors and the combination determined by other methods became minimal. Other hand, the joint-entropy at the multi-indicator vectors the most by choosing the balancing element and the uncertainty at the overall situation was the least in this method. Thus, the optimization model about the weight vectors combination was constructed. An example is also given to demonstrate the precision and reliability and validity of the method.
Information measures based on fractional calculus

Shiwei Yu,T.-Z. Huang,Xiaoyun Liu,Wufan Chen

DOI: https://doi.org/10.1016/j.ipl.2012.08.019

IF: 0.851

2012-01-01

Information Processing Letters

Abstract:In this letter, fractional calculus is used to propose the fractional entropy (FE) and the fractional mutual information (FMI) as the new forms of the information measure in a generalized Euclidean metric space. Being position-related and causal, FE and FMI are natural extensions and more generalized forms of Shannon entropy and mutual information, respectively.
General weighted information and relative information generating functions with properties

Shital Saha,Suchandan Kayal

2024-06-01

Abstract:In this work, we propose two information generating functions: general weighted information and relative information generating functions, and study their properties. { It is shown that the general weighted information generating function (GWIGF) is shift-dependent and can be expressed in terms of the weighted Shannon entropy. The GWIGF of a transformed random variable has been obtained in terms of the GWIGF of a known distribution. Several bounds of the GWIGF have been proposed. We have obtained sufficient conditions under which the GWIGFs of two distributions are comparable. Further, we have established a connection between the weighted varentropy and varentropy with proposed GWIGF. An upper bound for GWIGF of the sum of two independent random variables is derived. The effect of general weighted relative information generating function (GWRIGF) for two transformed random variables under strictly monotone functions has been studied. } Further, these information generating functions are studied for escort, generalized escort and mixture distributions. {Specially, we propose weighted $\beta$-cross informational energy and establish a close connection with GWIGF for escort distribution.} The residual versions of the newly proposed generating functions are considered and several similar properties have been explored. A non-parametric estimator of the residual general weighted information generating function is proposed. A simulated data set and two real data sets are considered for the purpose of illustration. { Finally, we have compared the non-parametric approach with a parametric approach in terms of the absolute bias and mean squared error values.}

Statistics Theory
Entropic Uncertainty Relations for General Symmetric Informationally Complete Positive Operator-Valued Measures and Mutually Unbiased Measurements

Shan Huang,Zeng-Bing Chen,Shengjun Wu

DOI: https://doi.org/10.1103/physreva.103.042205

2021-01-01

Abstract:We construct inequalities between Renyi alpha-entropy and the indexes of coincidence of probability distributions, based on which we obtain improved state-dependent entropic uncertainty relations for general symmetric informationally complete positive operator-valued measures (SIC-POVMs) and mutually unbiased measurements (MUMs). We show that our uncertainty relations for general SIC-POVMs and MUMs can be tight for sufficiently mixed states, and, moreover, comparisons to the numerically optimal results are made via information diagrams.
A simple proof of the entropy-power inequality

S. Verdu,D. Guo

DOI: https://doi.org/10.1109/TIT.2006.872978

IF: 2.5

2006-01-01

IEEE Transactions on Information Theory

Abstract:This correspondence gives a simple proof of Shannon's entropy-power inequality (EPI) using the relationship between mutual information and minimum mean-square error (MMSE) in Gaussian channels.
Mutual Information is Copula Entropy

Ma Jian,Sun Zengqi

DOI: https://doi.org/10.1016/s1007-0214(11)70008-6

2011-01-01

Abstract:Mutual information (MI) is a basic concept in information theory. Therefore, estimates of the MI are fundamentally important in most information theory applications. This paper provides a new way of understanding and estimating the MI using the copula function. First, the entropy of the copula, named the copula entropy, is defined as a measure of the dependence uncertainty represented by the copula function and then the MI is shown to be equivalent to the negative copula entropy. With this equivalence, the MI can be estimated by first estimating the empirical copula and then estimating the entropy of the empirical copula. Thus, the MI estimate is an estimation of the entropy, which reduces the complexity and computational requirements. Tests show that the method is more effective than the traditional method.
A New Extremal Entropy Inequality With Applications

Hon Fah Chong,Ying-Chang Liang

DOI: https://doi.org/10.1109/ISIT.2013.6620298

2013-01-01

Abstract:Liu et al. proved an extremal entropy inequality using a vector generalization of the Costa entropy-power inequality (EPI). The generalized Costa EPI was proved, in turn, using a perturbation approach via a fundamental relationship between the derivative of mutual information and the minimum mean-square error (MMSE) estimate in linear vector Gaussian channels. In this paper, we consider two new variations of the (Liu et al.) extremal entropy inequality. Instead of employing perturbation approaches, we employ a new method recently introduced by Geng and Nair, which was used to resolve the capacity region of the Gaussian MIMO broadcast channel (BC) with common and private messages. As an application, we use one of the extremal entropy inequalities to prove the capacity region of a class of reversely degraded Gaussian MIMO BC with three users and three-degraded message sets.
Effectiveness of Entropy Weight Method in Decision-Making

Yuxin Zhu,Dazuo Tian,Feng Yan

DOI: https://doi.org/10.1155/2020/3564835

IF: 1.43

2020-03-26

Mathematical Problems in Engineering

Abstract:Entropy weight method (EWM) is a commonly used weighting method that measures value dispersion in decision-making. The greater the degree of dispersion, the greater the degree of differentiation, and more information can be derived. Meanwhile, higher weight should be given to the index, and vice versa. This study shows that the rationality of the EWM in decision-making is questionable. One example is water source site selection, which is generated by Monte Carlo Simulation. First, too many zero values result in the standardization result of the EWM being prone to distortion. Subsequently, this outcome will lead to immense index weight with low actual differentiation degree. Second, in multi-index decision-making involving classification, the classification degree can accurately reflect the information amount of the index. However, the EWM only considers the numerical discrimination degree of the index and ignores rank discrimination. These two shortcomings indicate that the EWM cannot correctly reflect the importance of the index weight, thus resulting in distorted decision-making results.

engineering, multidisciplinary,mathematics, interdisciplinary applications
MIWE: detecting the critical states of complex biological systems by the mutual information weighted entropy

Yuke Xie,Xueqing Peng and Peiluan Li

DOI: https://doi.org/10.1186/s12859-024-05667-z

IF: 3.307

2024-01-28

BMC Bioinformatics

Abstract:Complex biological systems often undergo sudden qualitative changes during their dynamic evolution. These critical transitions are typically characterized by a catastrophic progression of the system. Identifying the critical point is critical to uncovering the underlying mechanisms of complex biological systems. However, the system may exhibit minimal changes in its state until the critical point is reached, and in the face of high throughput and strong noise data, traditional biomarkers may not be effective in distinguishing the critical state. In this study, we propose a novel approach, mutual information weighted entropy (MIWE), which uses mutual information between genes to build networks and identifies critical states by quantifying molecular dynamic differences at each stage through weighted differential entropy. The method is applied to one numerical simulation dataset and four real datasets, including bulk and single-cell expression datasets. The critical states of the system can be recognized and the robustness of MIWE method is verified by numerical simulation under the influence of different noises. Moreover, we identify two key transcription factors (TFs), CREB1 and CREB3, that regulate downstream signaling genes to coordinate cell fate commitment. The dark genes in the single-cell expression datasets are mined to reveal the potential pathway regulation mechanism.

biochemical research methods,biotechnology & applied microbiology,mathematical & computational biology
A Novel Dual of Shannon Information and Weighting Scheme

Arthur Jun Zhang

2023-04-25

Abstract:Shannon Information theory has achieved great success in not only communication technology where it was originally developed for but also many other science and engineering fields such as machine learning and artificial intelligence. Inspired by the famous weighting scheme TF-IDF, we discovered that information entropy has a natural dual. We complement the classical Shannon information theory by proposing a novel quantity, namely troenpy. Troenpy measures the certainty, commonness and similarity of the underlying distribution. To demonstrate its usefulness, we propose a troenpy based weighting scheme for document with class labels, namely positive class frequency (PCF). On a collection of public datasets we show the PCF based weighting scheme outperforms the classical TF-IDF and a popular Optimal Transportation based word moving distance algorithm in a kNN setting. We further developed a new odds-ratio type feature, namely Expected Class Information Bias(ECIB), which can be regarded as the expected odds ratio of the information quantity entropy and troenpy. In the experiments we observe that including the new ECIB features and simple binary term features in a simple logistic regression model can further significantly improve the performance. The simple new weighting scheme and ECIB features are very effective and can be computed with linear order complexity.

Computation and Language,Information Retrieval,Information Theory,Machine Learning

Exponential weighted entropy and exponential weighted mutual information

Exponential Weighted Entropy and Exponential Weighted Mutual Information.

Weighted entropy: basic inequalities

A Modified Belief Entropy in Dempster-Shafer Framework

Basic inequalities for weighted entropies

Information Theoretic Proofs of Entropy Power Inequalities

Conditional Deng Entropy, Joint Deng Entropy and Generalized Mutual Information

Adaptive Exponential Weighted Mutual Information As a Similarity Measure for Medical Image Registration

Generalized information entropy and generalized information dimension

Shannon Shakes Hands with Chernoff: Big Data Viewpoint on Channel Information Measures.

Mutual Information Multinomial Estimation

Combination Weighting Approach Based on Deviation Function and Joint-Entropy

Information measures based on fractional calculus

General weighted information and relative information generating functions with properties

Entropic Uncertainty Relations for General Symmetric Informationally Complete Positive Operator-Valued Measures and Mutually Unbiased Measurements

A simple proof of the entropy-power inequality

Mutual Information is Copula Entropy

A New Extremal Entropy Inequality With Applications

Effectiveness of Entropy Weight Method in Decision-Making

MIWE: detecting the critical states of complex biological systems by the mutual information weighted entropy

A Novel Dual of Shannon Information and Weighting Scheme