Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations

Max Schemmer,Niklas Kühl,Carina Benz,Andrea Bartos,Gerhard Satzger

DOI: https://doi.org/10.1145/3581641.3584066

2023-04-13

Abstract:AI advice is becoming increasingly popular, e.g., in investment and medical treatment decisions. As this advice is typically imperfect, decision-makers have to exert discretion as to whether actually follow that advice: they have to "appropriately" rely on correct and turn down incorrect advice. However, current research on appropriate reliance still lacks a common definition as well as an operational measurement concept. Additionally, no in-depth behavioral experiments have been conducted that help understand the factors influencing this behavior. In this paper, we propose Appropriateness of Reliance (AoR) as an underlying, quantifiable two-dimensional measurement concept. We develop a research model that analyzes the effect of providing explanations for AI advice. In an experiment with 200 participants, we demonstrate how these explanations influence the AoR, and, thus, the effectiveness of AI advice. Our work contributes fundamental concepts for the analysis of reliance behavior and the purposeful design of AI advisors.

Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to achieve appropriate reliance (AR) on AI suggestions when human decision - makers use AI suggestions. As AI provides suggestions more and more frequently in fields such as investment and medical treatment, decision - makers need to judge when they should trust and follow these suggestions and when they should stick to their own judgment. However, current research on AR has problems such as unclear definitions, lack of measurement concepts, and insufficient research on influencing factors. Specifically, the paper focuses on the following aspects:

1. **Ambiguity of the AR concept**: In current research, AR is used to describe both the binary target state (that is, AR is either achieved or not achieved) and a metric indicating the degree of adaptation. This ambiguity hinders in - depth understanding and effective measurement of AR.

2. **Lack of measurement concept for AR**: Although AR is considered an important condition for the effective use of AI, there is currently no unified measurement concept to quantify the degree of AR.

3. **The impact of explanation on AR**: Existing literature is controversial about whether explanation can support AR. Some studies show that explanation helps to improve AR, while other studies find that explanation may lead to blind trust in AI suggestions.

To solve these problems, the paper proposes the "appropriateness of reliance (AoR)" - a two - dimensional measurement concept, aiming to describe and measure reliance behavior. AoR is measured based on relative frequency, including correctly covering wrong AI suggestions (self - reliance) and following correct AI suggestions (AI - reliance). In addition, the paper also analyzes the impact of providing explanations on AoR through experiments, thereby providing theoretical basis and technical guidance for the conceptualization and practical application of AR.

In summary, through the introduction of the AoR concept and experimental verification, this paper aims to fill the gaps in AR research and provide theoretical support for designing more effective AI advisor systems.