Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models

Marvin Pafla,Kate Larson,Mark Hancock

DOI: https://doi.org/10.1145/3613904.3642934

2024-04-11

Abstract:The field of eXplainable artificial intelligence (XAI) has produced a plethora of methods (e.g., saliency-maps) to gain insight into artificial intelligence (AI) models, and has exploded with the rise of deep learning (DL). However, human-participant studies question the efficacy of these methods, particularly when the AI output is wrong. In this study, we collected and analyzed 156 human-generated text and saliency-based explanations collected in a question-answering task (N=40) and compared them empirically to state-of-the-art XAI explanations (integrated gradients, conservative LRP, and ChatGPT) in a human-participant study (N=136). Our findings show that participants found human saliency maps to be more helpful in explaining AI answers than machine saliency maps, but performance negatively correlated with trust in the AI model and explanations. This finding hints at the dilemma of AI errors in explanation, where helpful explanations can lead to lower task performance when they support wrong AI predictions.

Human-Computer Interaction,Artificial Intelligence

What problem does this paper attempt to address?

This paper discusses the dilemma of AI error explanations, with a particular focus on the explanatory effectiveness of large-scale language models. The study compares the performance of human and machine-generated saliency maps and textual explanations in question-answering tasks through experiments with human participants. The results indicate that although human-generated saliency maps are considered more helpful, trust and satisfaction with AI are negatively correlated with task performance. The paper reveals a paradox where AI predictions that facilitate explanations may lead to lower task performance, especially when these explanations support erroneous predictions. The study also finds that in the case of incorrect answers, no explanation performs even better than certain types of explanations. The paper highlights the danger of confirmation bias in explanations and provides recommendations for designers and researchers on the use of saliency maps to mitigate this bias.