Abstract:Design time uncertainty poses an important challenge when developing a self-adaptive system. As an example, defining how the system should adapt when facing a new environment state, requires understanding the precise effect of an adaptation, which may not be known at design time. Online reinforcement learning, i.e., employing reinforcement learning (RL) at runtime, is an emerging approach to realizing self-adaptive systems in the presence of design time uncertainty. By using Online RL, the self-adaptive system can learn from actual operational data and leverage feedback only available at runtime. Recently, Deep RL is gaining interest. Deep RL represents learned knowledge as a neural network whereby it can generalize over unseen inputs, as well as handle continuous environment states and adaptation actions. A fundamental problem of Deep RL is that learned knowledge is not explicitly represented. For a human, it is practically impossible to relate the parametrization of the neural network to concrete RL decisions and thus Deep RL essentially appears as a black box. Yet, understanding the decisions made by Deep RL is key to (1) increasing trust, and (2) facilitating debugging. Such debugging is especially relevant for self-adaptive systems, because the reward function, which quantifies the feedback to the RL algorithm, must be defined by developers. The reward function must be explicitly defined by developers, thus introducing a potential for human error. To explain Deep RL for self-adaptive systems, we enhance and combine two existing explainable RL techniques from the machine learning literature. The combined technique, XRL-DINE, overcomes the respective limitations of the individual techniques. We present a proof-of-concept implementation of XRL-DINE, as well as qualitative and quantitative results of applying XRL-DINE to a self-adaptive system exemplar.

Explainable Reinforcement Learning: A Survey and Comparative Review

A Survey of Explainable Reinforcement Learning

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Explainable Reinforcement Learning: Basic Problems Exploration and Method Survey

Explainable AI and Reinforcement Learning-A Systematic Review of Current Approaches and Trends

A survey on interpretable reinforcement learning

Experiential Explanations for Reinforcement Learning

A User Study on Explainable Online Reinforcement Learning for Adaptive Systems

Explainable Reinforcement Learning via Model Transforms

Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes

Utilizing Explainability Techniques for Reinforcement Learning Model Assurance

Explainable Deep Reinforcement Learning: State of the Art and Challenges

Explainable Recommendation: A Survey and New Perspectives

A Survey on Causal Reinforcement Learning

Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Explaining Online Reinforcement Learning Decisions of Self-Adaptive Systems

Causal State Distillation for Explainable Reinforcement Learning

Learning impartial policies for sequential counterfactual explanations using Deep Reinforcement Learning