A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty

Parvin Malekzadeh,Ming Hou,Konstantinos N. Plataniotis

DOI: https://doi.org/10.1109/ICASSP49357.2023.10095594

2024-01-06

Abstract:Exploration is a significant challenge in practical reinforcement learning (RL), and uncertainty-aware exploration that incorporates the quantification of epistemic and aleatory uncertainty has been recognized as an effective exploration strategy. However, capturing the combined effect of aleatory and epistemic uncertainty for decision-making is difficult. Existing works estimate aleatory and epistemic uncertainty separately and consider the composite uncertainty as an additive combination of the two. Nevertheless, the additive formulation leads to excessive risk-taking behavior, causing instability. In this paper, we propose an algorithm that clarifies the theoretical connection between aleatory and epistemic uncertainty, unifies aleatory and epistemic uncertainty estimation, and quantifies the combined effect of both uncertainties for a risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution whose parameters are random variables encoding epistemic uncertainty. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.

Machine Learning

What problem does this paper attempt to address?

The paper aims to address a significant challenge in exploration strategies within Reinforcement Learning (RL), specifically how to effectively combine epistemic uncertainty and aleatoric uncertainty during the decision-making process. Specifically:

- **Problem Background**: Current RL algorithms face the issue of low sample efficiency in practical applications, especially when data collection is costly or risky. To improve this issue, a promising approach is to adopt uncertainty-aware exploration strategies, which leverage the uncertainties in the agent and the environment for more effective exploration.

- **Issues with Existing Methods**: Current methods typically estimate epistemic uncertainty and aleatoric uncertainty separately and then simply add them to form composite uncertainty. However, this additive combination can lead to overly risky behavior, resulting in system instability.

- **Contribution of the Paper**: This paper proposes a new algorithm that aims to establish a theoretical connection between epistemic uncertainty and aleatoric uncertainty, unifying the estimation of these two types of uncertainties and quantifying their combined effect to achieve risk-sensitive exploration. The method is based on an innovative extension of the distributed reinforcement learning framework, modeling belief distributions through a mixture of Dirac functions and utilizing Moment-Generating Functions (MGFs) to extract features, ultimately achieving an exploration strategy that considers composite uncertainty.

In summary, the goal of this paper is to improve exploration efficiency and stability in reinforcement learning by integrating epistemic uncertainty and aleatoric uncertainty through a novel approach.