Abstract:Automated decision-making systems are becoming increasingly ubiquitous, which creates an immediate need for their interpretability and explainability. However, it remains unclear whether users know what insights an explanation offers and, more importantly, what information it lacks. To answer this question we conducted an online study with 200 participants, which allowed us to assess explainees' ability to realise explicated information -- i.e., factual insights conveyed by an explanation -- and unspecified information -- i.e, insights that are not communicated by an explanation -- across four representative explanation types: model architecture, decision surface visualisation, counterfactual explainability and feature importance. Our findings uncover that highly comprehensible explanations, e.g., feature importance and decision surface visualisation, are exceptionally susceptible to misinterpretation since users tend to infer spurious information that is outside of the scope of these explanations. Additionally, while the users gauge their confidence accurately with respect to the information explicated by these explanations, they tend to be overconfident when misinterpreting the explanations. Our work demonstrates that human comprehension can be a double-edged sword since highly accessible explanations may convince users of their truthfulness while possibly leading to various misinterpretations at the same time. Machine learning explanations should therefore carefully navigate the complex relation between their full scope and limitations to maximise understanding and curb misinterpretation.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to explore users' understanding of machine - learning explanations and their misunderstandings. Specifically, the researchers are concerned with: 1. **Whether users can be aware of the information explicitly conveyed in the explanations**: that is, the factual insights provided by the explanations. 2. **Whether users can identify the information not explicitly conveyed in the explanations**: that is, the insights not covered by the explanations, which may be wrongly inferred by users. 3. **How users' understanding and misunderstandings affect their confidence in the explanations**: The study found that users tend to be over - confident when they misunderstand the explanations. Through online experiments, the researchers evaluated the impact of four representative explanation methods (model architecture, decision - surface visualization, counterfactual explanations, and feature importance) on 200 participants. The research results show that highly understandable explanations (such as feature importance and decision - surface visualization) are more likely to be misunderstood by users, and users tend to be over - confident when they misunderstand the explanations. This indicates that although easy - to - understand explanations help improve users' understanding, they may also lead users to over - interpret the information not explicitly conveyed, thus causing misunderstandings. ### Main contributions 1. **Introducing the concept of "information not explicitly conveyed"**: The researchers defined the information misread by users based on reliable machine - learning explanations and emphasized the importance of evaluating the explicit and non - explicit information of the explanations. 2. **Revealing the double - edged - sword effect of understanding**: Highly understandable explanations may be easily misunderstood at the same time because users are not sensitive enough to the limitations of "easy - to - understand" explanations. 3. **Designing a flexible evaluation framework**: The researchers provided a reusable framework for evaluating the explicit and non - explicit information of the explanations, thus helping to identify and reduce the misleading of the explanations. ### Research background With the popularization of automated decision - making systems, explanation and interpretability are becoming more and more important. However, it is still unclear whether users can correctly understand the information provided by the explanations and whether they are aware of the limitations of the explanations. This paper, through systematic research, reveals the common misunderstandings of users in understanding machine - learning explanations and provides important references for future design and applications.

Comprehension Is a Double-Edged Sword: Over-Interpreting Unspecified Information in Intelligible Machine Learning Explanations

Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations

An Evaluation of the Human-Interpretability of Explanation

Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users' Information Processing

Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence

"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations

Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration

Machine Explanations and Human Understanding

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models

Explaining Explanations: An Overview of Interpretability of Machine Learning

Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting

Stop ordering machine learning algorithms by their explainability! A user-centered investigation of performance and explainability

How Level of Explanation Detail Affects Human Performance in Interpretable Intelligent Systems: A Study on Explainable Fact Checking

Understanding the Effect of Algorithm Transparency of Model Explanations in Text-to-SQL Semantic Parsing

Explanation matters: An experimental study on explainable AI

On the Impact of Explanations on Understanding of Algorithmic Decision-Making

One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency

Explaining Explanations in AI

Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

"Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations