The Principle of Uncertain Maximum Entropy

Kenneth Bogert,Matthew Kothe
2024-09-11
Abstract:The principle of maximum entropy is a well-established technique for choosing a distribution that matches available information while minimizing bias. It finds broad use across scientific disciplines and in machine learning. However, the principle as defined by is susceptible to noise and error in observations. This forces real-world practitioners to use relaxed versions of the principle in an ad hoc way, negatively impacting interpretation. To address this situation, we present a new principle we call uncertain maximum entropy that generalizes the classic principle and provides interpretable solutions irrespective of the observational methods in use. We introduce a convex approximation and expectation-maximization based algorithm for finding solutions to our new principle. Finally, we contrast this new technique with two simpler generally applicable solutions theoretically and experimentally show our technique provides superior accuracy.
Information Theory,Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
The paper attempts to address the problem of how to improve the application of the maximum entropy principle in the presence of observational noise. Specifically, the paper proposes a new Principle of Uncertain Maximum Entropy to overcome the shortcomings of the traditional maximum entropy principle when dealing with noisy observational data. The traditional maximum entropy principle assumes that observational data is noise-free, which is not realistic in practical applications. Therefore, the researchers propose a general method to handle arbitrary observational errors and provide interpretable solutions. The paper addresses this issue by introducing a new nonlinear programming problem that combines feature expectation constraints with observational functions. Additionally, the paper proposes a convex approximation method and a solution method based on the Expectation-Maximization algorithm. Experimental results show that the new method outperforms several existing simple methods, including the Most-Likely-x method and the two-stage maximum entropy method (MaxEnt-MaxEnt), in various scenarios. Overall, the paper aims to provide a more robust and accurate implementation of the maximum entropy principle that remains effective when dealing with noisy observational data.