Abstract:Details of the designs and mechanisms in support of human-AI collaboration must be considered in the real-world fielding of AI technologies. A critical aspect of interaction design for AI-assisted human decision making are policies about the display and sequencing of AI inferences within larger decision-making workflows. We have a poor understanding of the influences of making AI inferences available before versus after human review of a diagnostic task at hand. We explore the effects of providing AI assistance at the start of a diagnostic session in radiology versus after the radiologist has made a provisional decision. We conducted a user study where 19 veterinary radiologists identified radiographic findings present in patients' X-ray images, with the aid of an AI tool. We employed two workflow configurations to analyze (i) anchoring effects, (ii) human-AI team diagnostic performance and agreement, (iii) time spent and confidence in decision making, and (iv) perceived usefulness of the AI. We found that participants who are asked to register provisional responses in advance of reviewing AI inferences are less likely to agree with the AI regardless of whether the advice is accurate and, in instances of disagreement with the AI, are less likely to seek the second opinion of a colleague. These participants also reported the AI advice to be less useful. Surprisingly, requiring provisional decisions on cases in advance of the display of AI inferences did not lengthen the time participants spent on the task. The study provides generalizable and actionable insights for the deployment of clinical AI tools in human-in-the-loop systems and introduces a methodology for studying alternative designs for human-AI collaboration. We make our experimental platform available as open source to facilitate future research on the influence of alternate designs on human-AI workflows.

Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making

Uncalibrated Models Can Improve Human-AI Collaboration

Three Challenges for AI-Assisted Decision-Making

Towards Effective Human-AI Decision-Making: The Role of Human Learning in Appropriate Reliance on AI Advice

Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork

On the Effect of Information Asymmetry in Human-AI Teams

How Being Outvoted by AI Teammates Impacts Human-AI Collaboration

Designing Algorithmic Recommendations to Achieve Human-AI Complementarity

Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making

Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction

"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making

Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology

Human Designers' Dynamic Confidence and Decision-Making When Working with More than One AI

Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions

Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging

Improving Human-AI Collaboration With Descriptions of AI Behavior

Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence

Designing Human and Generative AI Collaboration

Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosis

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

The Role of Heuristics and Biases During Complex Choices with an AI Teammate