How to Measure Human-AI Prediction Accuracy in Explainable AI Systems

Sujay Koujalgi,Andrew Anderson,Iyadunni Adenuga,Shikha Soneji,Rupika Dikkala,Teresita Guzman Nader,Leo Soccio,Sourav Panda,Rupak Kumar Das,Margaret Burnett,Jonathan Dodge

2024-08-24

Abstract:Assessing an AI system's behavior-particularly in Explainable AI Systems-is sometimes done empirically, by measuring people's abilities to predict the agent's next move-but how to perform such measurements? In empirical studies with humans, an obvious approach is to frame the task as binary (i.e., prediction is either right or wrong), but this does not scale. As output spaces increase, so do floor effects, because the ratio of right answers to wrong answers quickly becomes very small. The crux of the problem is that the binary framing is failing to capture the nuances of the different degrees of "wrongness." To address this, we begin by proposing three mathematical bases upon which to measure "partial wrongness." We then uses these bases to perform two analyses on sequential decision-making domains: the first is an in-lab study with 86 participants on a size-36 action space; the second is a re-analysis of a prior study on a size-4 action space. Other researchers adopting our operationalization of the prediction task and analysis methodology will improve the rigor of user studies conducted with that task, which is particularly important when the domain features a large output space.

Human-Computer Interaction,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to measure the accuracy of human predictions of AI in interpretable artificial intelligence systems. Specifically, existing methods usually evaluate the ability of human participants to predict AI behavior through a binary framework (i.e., prediction is correct or incorrect), but this method has limitations when the output space is large because it cannot capture different degrees of "error". As the output space increases, the ratio of correct answers to incorrect answers becomes very small, resulting in insufficient information in the evaluation results under the binary framework and being prone to Type II errors (i.e., failing to detect differences or associations that actually exist). Therefore, this paper proposes a new method, aiming to improve this evaluation method by introducing the concept of "partial error", thereby measuring the accuracy of human predictions of AI more accurately. This method can not only improve the accuracy of evaluation but also enhance the ability to evaluate the transparency of AI systems, especially in cases where the output space is large.

How to Measure Human-AI Prediction Accuracy in Explainable AI Systems

Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction

Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance

Human Learning about AI Performance

Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making

Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models

Explainable AI improves task performance in human-AI collaboration

Metrics for Explainable AI: Challenges and Prospects

The Impact of Imperfect XAI on Human-AI Decision-Making

Evaluating the Impact of Human Explanation Strategies on Human-AI Visual Decision-Making

Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations

Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems

Aligning Judgment Using Task Context and Explanations to Improve Human-Recommender System Performance

Training on the Test Set: Mapping the System-Problem Space in AI

How Level of Explanation Detail Affects Human Performance in Interpretable Intelligent Systems: A Study on Explainable Fact Checking

On Evaluating Explanation Utility for Human-AI Decision Making in NLP

A Measure of Explanatory Effectiveness

Explainability does not mitigate the negative impact of incorrect AI advice in a personnel selection task

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

The Who in XAI: How AI Background Shapes Perceptions of AI Explanations

Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making