An information fractal dimensional relative entropy

Jingyou Wu

DOI: https://doi.org/10.1063/5.0189038

IF: 1.697

2024-02-01

AIP Advances

Abstract:Shannon entropy is used to measure information uncertainty, while the information dimension is used to measure information complexity. Given two probability distributions, the difference can be measured by relative entropy. However, the existing relative entropy does not consider the effect of information dimension. To improve the existing entropy, a new relative entropy is presented in this paper. The information fractal dimension is considered in the proposed relative entropy. The new relative entropy is more generalized than the initial relative entropy. When dimension is not considered, it will degenerate to the initial relative entropy. Another interesting point is that the new relative entropy may have negative values when calculating. The physical meaning is still under exploration. Finally, some application examples are provided to exemplify the utilization of the proposed relative entropy.

materials science, multidisciplinary,physics, applied,nanoscience & nanotechnology

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that when the existing relative entropy measures the differences between different probability distributions, it does not take into account the influence of the information dimension. Specifically, the traditional relative entropy can only measure the differences between two probability distributions, but cannot reflect the information complexity or roughness level of these distributions. To improve this, the author proposes a new relative entropy, namely Information Fractal Dimensional Relative Entropy. This new relative entropy combines the information dimension, so it can more comprehensively measure the differences between probability distributions.

### Main contributions:
1. **Introduction of information dimension**: The newly proposed relative entropy not only considers the differences in probability distributions, but also takes into account the influence of the information dimension, making the calculation of relative entropy more comprehensive and accurate.
2. **Symmetry and non - negativity**: The optimized relative entropy has symmetry, that is, \( E_{\text{odr}}(P \| Q) = E_{\text{odr}}(Q \| P) \), and remains non - negative in most cases.
3. **Possible negative values**: The new relative entropy may produce negative values in some cases, which is impossible in the traditional relative entropy. Although the physical meaning of negative values is still being explored, this provides a new direction for research.
4. **Application examples**: Through multiple application examples, the effectiveness and superiority of the new relative entropy in practical problems are demonstrated, especially in scenarios involving changes in the information dimension.

### Mathematical expressions:
- **Shannon entropy**:
\[
H_s = -\sum_{i = 1}^{n} p_i \log p_i
\]
- **Rényi entropy**:
\[
H_\alpha(X) = \frac{1}{1 - \alpha} \log \left( \sum_{i = 1}^{n} p_i^\alpha \right)
\]
- **Information dimension**:
\[
D_r = \lim_{m \to 0} \frac{\log \left( \sum_{i = 1}^{n} p_i^k \right)}{\log m}
\]
When \( k \to 1 \), the information dimension can be expressed as:
\[
D_r = \lim_{n \to \infty} \frac{-\sum_{i = 1}^{n} p_i \log p_i}{\log n}
\]
- **Traditional relative entropy**:
\[
E_r(P \| Q) = \sum_{i = 1}^{n} p_i \log \frac{p_i}{q_i}
\]
- **Newly proposed relative entropy**:
\[
E_{\text{dr}}(P \| Q) = \sum_{i = 1}^{n} p_i^{D_P} \log \frac{p_i^{D_P}}{q_i^{D_Q}}
\]
Where \( D_P \) and \( D_Q \) represent the information dimensions of \( P \) and \( Q \) respectively:
\[
D_P = \frac{-\sum_{i = 1}^{n} p_i \log p_i}{\log n}, \quad D_Q = \frac{-\sum_{i = 1}^{n} q_i \log q_i}{\log n}
\]
- **Optimized relative entropy**:
\[
E_{\text{odr}}(P \| Q) = \frac{E_{\text{dr}}(P \| Q) + E_{\text{dr}}(Q \| P)}{2}
\]

Through these improvements, the new relative entropy can provide more accurate measurements in more complex scenarios, especially in applications involving changes in the information dimension.