Abstract:In response to the demand for Explainable Artificial Intelligence (XAI), we investigate the use of Large Language Models (LLMs) to transform ML explanations into natural, human-readable narratives. Rather than directly explaining ML models using LLMs, we focus on refining explanations computed using existing XAI algorithms. We outline several research directions, including defining evaluation metrics, prompt design, comparing LLM models, exploring further training methods, and integrating external data. Initial experiments and user study suggest that LLMs offer a promising way to enhance the interpretability and usability of XAI.

What problem does this paper attempt to address?

This paper discusses how to enhance the interpretability and usability of machine learning (ML) explanations using large language models (LLMs), particularly by transforming existing ML explanations into natural and readable narratives. The research focuses on improving explanations generated by existing explanation algorithms rather than directly using LLMs to explain ML models. The authors propose several research directions: 1. Define evaluation metrics: Formal evaluation criteria are needed to compare narrative explanations generated by different models and techniques. 2. Instruction design: Investigate the ability of LLMs to generate explanatory narratives in zero-shot scenarios and explore how to use customized instructions to guide the model to generate nuanced and contextually diverse explanations. 3. Compare LLMs: Compare different LLMs in terms of their effectiveness in generating narrative explanations to understand their strengths and weaknesses. 4. Further training and fine-tuning: Improve the explanations generated by LLMs through training methods such as model fine-tuning. 5. Incorporate external data training: Utilize model training data and information from related domains, such as guidelines or textbooks, to generate more context-aware explanations. Preliminary experiments and user studies indicate the potential of LLMs in enhancing explainability and usability of explainable artificial intelligence (XAI). In the paper, the authors performed zero-shot conversion experiments on SHAP explanations using GPT-3.5 and GPT-4, and evaluated the generated narratives by designing different instructions. The results show that GPT-4 outperforms GPT-3.5 in terms of correctness, completeness, and context awareness, but GPT-3.5's responses are more concise and fluent. Furthermore, a small user study found that most participants preferred narrative-based explanations, considering them easier to understand and more informative. These findings support the potential benefits of using LLMs to generate narrative explanations. Future work includes further research on the aforementioned directions, such as fine-tuning methods, exploring more LLMs, and integrating training data and external information to create richer explanations. The goal is to enhance the quality of narrative explanations, promote transparency, interpretability, and usability of AI systems, and thereby enhance user trust and understanding of AI technology.

LLMs for XAI: Future Directions for Explaining Explanations

Explingo: Explaining AI Predictions using Large Language Models

XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models

Evaluating Explanations Through LLMs: Beyond Traditional User Studies

XAI for All: Can Large Language Models Simplify Explainable AI?

Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era

XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs

Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models

From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers

From Understanding to Utilization: A Survey on Explainability for Large Language Models

From Feature Importance to Natural Language Explanations Using LLMs with RAG

How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives

Towards Uncovering How Large Language Model Works: An Explainability Perspective

From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI

Argumentative Large Language Models for Explainable and Contestable Decision-Making

Explaining Explanations: An Overview of Interpretability of Machine Learning

Explainability for Large Language Models: A Survey

Properties and Challenges of LLM-Generated Explanations

Interpretable Narrative Explanation for ML Predictors with LP: A Case Study for XAI

A survey on XAI and natural language explanations

Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement