Abstract:Intelligent agents such as robots are increasingly deployed in real-world, safety-critical settings. It is vital that these agents are able to explain the reasoning behind their decisions to human counterparts, however, their behavior is often produced by uninterpretable models such as deep neural networks. We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions, agnostic to the underlying model representation. We show how a compact representation of the agent's behavior can be learned and used to produce plausible explanations with minimal hallucination while affording user interaction with a pre-trained large language model. Through user studies and empirical experiments, we show that our approach generates explanations as helpful as those generated by a human domain expert while enabling beneficial interactions such as clarification and counterfactual queries.

What problem does this paper attempt to address?

The paper attempts to address the problem of providing natural language explanations of an agent's (e.g., a robot's) behavior when it is generated by an inexplicable model (such as a deep neural network) to human users. Specifically, the paper proposes a method to generate natural language explanations of an agent's behavior solely by observing states and actions, without relying on the specific representation of the underlying model. This method aims to reduce hallucinations in explanations (i.e., generating false information) while allowing users to interact with pre-trained large-scale language models for deeper understanding. The key contributions of the paper include: 1. **Behavior Representation (BR)**: By refining the agent's policy into an interpretable decision tree and extracting decision paths in specific states, forming a locally interpretable behavior model. 2. **Generating Explanations Using Large-Scale Language Models**: By injecting the behavior representation into the prompts of large-scale language models to generate natural language explanations, this method can significantly reduce hallucinations while producing useful and reasonable explanations. 3. **Interactive Explanations**: Allowing users to ask questions for further clarification or hypothetical queries, thereby enhancing the transparency and comprehensibility of the explanations. User studies and experiments demonstrate that the explanations generated by this method are comparable to those generated by human experts in helping users understand agent behavior, and in some cases, even more preferred, especially when the agent's policy is suboptimal, allowing users to gain more assistance through interaction. Additionally, the method performs excellently in reducing hallucinations.

Explaining Agent Behavior with Large Language Models

Towards Uncovering How Large Language Model Works: An Explainability Perspective

Large Language Models Cannot Explain Themselves

Explainability for Large Language Models: A Survey

Explingo: Explaining AI Predictions using Large Language Models

From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers

Natural Language Counterfactual Explanations for Graphs Using Large Language Models

Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration

Explanation of Reinforcement Learning Model in Dynamic Multi-Agent System

Explanation of Reinforcement Learning Model in Dynamic Multi-Agent System

Mental Modeling of Reinforcement Learning Agents by Language Models

From Feature Importance to Natural Language Explanations Using LLMs with RAG

LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks

XAI for All: Can Large Language Models Simplify Explainable AI?

Rethinking Interpretability in the Era of Large Language Models

Automated Natural Language Explanation of Deep Visual Neurons with Large Models

Explaining Genetic Programming Trees using Large Language Models

Evaluating the Reliability of Self-Explanations in Large Language Models

Multi-Agent Strategy Explanations for Human-Robot Collaboration

Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement

Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects