Abstract:Understanding emerging behaviors of reinforcement learning (RL) agents may be difficult since such agents are often trained in complex environments using highly complex decision making procedures. This has given rise to a variety of approaches to explainability in RL that aim to reconcile discrepancies that may arise between the behavior of an agent and the behavior that is anticipated by an observer. Most recent approaches have relied either on domain knowledge that may not always be available, on an analysis of the agent's policy, or on an analysis of specific elements of the underlying environment, typically modeled as a Markov Decision Process (MDP). Our key claim is that even if the underlying model is not fully known (e.g., the transition probabilities have not been accurately learned) or is not maintained by the agent (i.e., when using model-free methods), the model can nevertheless be exploited to automatically generate explanations. For this purpose, we suggest using formal MDP abstractions and transforms, previously used in the literature for expediting the search for optimal policies, to automatically produce explanations. Since such transforms are typically based on a symbolic representation of the environment, they can provide meaningful explanations for gaps between the anticipated and actual agent behavior. We formally define the explainability problem, suggest a class of transforms that can be used for explaining emergent behaviors, and suggest methods that enable efficient search for an explanation. We demonstrate the approach on a set of standard benchmarks.

Explaining Reinforcement Learning to Mere Mortals: An Empirical Study

Experiential Explanations for Reinforcement Learning

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Explainable Reinforcement Learning Through a Causal Lens

Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

A User Study on Explainable Online Reinforcement Learning for Adaptive Systems

A Closer Look at Reward Decomposition for High-Level Robotic Explanations

Explainable Reinforcement Learning via Model Transforms

Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning

Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data

(When) Are Contrastive Explanations of Reinforcement Learning Helpful?

Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

Verbal Explanations for Deep Reinforcement Learning Neural Networks with Attention on Extracted Features.

Tell me why! Explanations support learning relational and causal structure

Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction

Incorporating Explanations to Balance the Exploration and Exploitation of Deep Reinforcement Learning.

Towards Explainable Reinforcement Learning Using Scoring Mechanism Augmented Agents

On Generating Explanations for Reinforcement Learning Policies: An Empirical Study

Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture

Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective.