On entropic convergence of algorithms in terms of domain partitions

Anatol Slissenko

DOI: https://doi.org/10.48550/arXiv.1605.01519

2016-05-05

Abstract:The paper describes an approach to measuring convergence of an algorithm to its result in terms of an entropy-like function of partitions of its inputs of a given length. The goal is to look at the algorithmic data processing from the viewpoint of information transformation, with a hope to better understand the work of algorithm, and maybe its complexity. The entropy is a measure of uncertainty, it does not correspond to our intuitive understanding of information. However, it is what we have in this area. In order to realize this approach we introduce a measure on the inputs of a given length based on the Principle of Maximal Uncertainty: all results should be equiprobable to the algorithm at the beginning. An algorithm is viewed as a set of events, each event is an application of a command. The commands are very basic. To measure the convergence we introduce a measure that is called entropic weight of events of the algorithm. The approach is illustrated by two examples.

Computational Complexity

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to quantify the speed at which an algorithm converges to the final result when processing data. Specifically, the author proposes an entropy - based method to measure the convergence speed of an algorithm under a given input length. By introducing a probability measure based on the Principle of Maximal Uncertainty and the entropic weight of events, the author hopes to gain a deeper understanding of the working mechanism and complexity of the algorithm from the perspective of information transformation.

### Background and Objectives of the Paper
1. **Problem Background**:
- When an algorithm processes input data, it gradually extracts information and tends towards the final result.
- Quantifying this information extraction process helps to gain a deeper understanding of the algorithm's complexity and design more efficient algorithms.
- Although there is no completely intuitive mathematical theory in information theory, entropy, as a measure of uncertainty, can be used to evaluate the progress of an algorithm's information processing.

2. **Research Objectives**:
- Introduce a probability measure and an entropy function to evaluate the speed at which an algorithm converges to the final result.
- Illustrate the application of this method through specific examples.

### Main Contributions
1. **Probability Measure**:
- Based on the Principle of Maximal Uncertainty, assume that all output results are equally probable at the beginning of the algorithm.
- Define a static probability measure that does not change during the execution of the algorithm.

2. **Entropy Weight**:
- Measure the degree of information extraction of the algorithm at each step through the entropy weight of events.
- The entropy weight is defined as \( D(E) = -\sum_{k} P(\hat{E} \cap \hat{F}_k) \log \frac{P(\hat{E} \cap \hat{F}_k)}{P(\hat{E})} \), where \( \hat{E} \) is the input set related to event \( E \), and \( \hat{F}_k \) is the input set corresponding to the \( k \) - th output value of function \( F \).

3. **Application Examples**:
- Through two specific algorithm examples (Sum over F2 and Maximal Prefix - Suffix), show how to use the entropy weight to evaluate the convergence speed of the algorithm.
- Analyze the behavior of these algorithms under different inputs, especially how they gradually reduce uncertainty and finally reach a definite result.

### Conclusions
- The proposed method provides a new perspective to understand and evaluate the convergence speed of algorithms.
- Through the entropy weight, the information extraction process of the algorithm at each step can be analyzed more finely.
- This method may be helpful in designing more efficient algorithms in practical applications, especially when dealing with complex data.

### Formula Presentation
- **Entropy Weight Formula**:
\[
D(E) = -\sum_{k} P(\hat{E} \cap \hat{F}_k) \log \frac{P(\hat{E} \cap \hat{F}_k)}{P(\hat{E})}
\]
where:
- \( P(\hat{E} \cap \hat{F}_k) \) is the probability of the intersection of the input set of event \( E \) and the input set corresponding to the \( k \) - th output value of function \( F \).
- \( P(\hat{E}) \) is the probability of the input set of event \( E \).

- **Conditional Probability**:
\[
P(\hat{E} \cap \hat{F}_k) = P(\hat{E}) \cdot P(\hat{F}_k \mid \hat{E})
\]

- **Derivative in Monotonicity Proof**:
\[
D'(S(t)) = -\sum_{k} \left( x_k' \log x_k + x_k' \right) + \left( \sum_{k} x_k' \right) \log \left( \sum_{k}