Abstract:Artificial Intelligence (AI) is increasingly used to build Decision Support Systems (DSS) across many domains. This paper describes a series of experiments designed to observe human response to different characteristics of a DSS such as accuracy and bias, particularly the extent to which participants rely on the DSS, and the performance they achieve. In our experiments, participants play a simple online game inspired by so-called "wildcat" (i.e., exploratory) drilling for oil. The landscape has two layers: a visible layer describing the costs (terrain), and a hidden layer describing the reward (oil yield). Participants in the control group play the game without receiving any assistance, while in treatment groups they are assisted by a DSS suggesting places to drill. For certain treatments, the DSS does not consider costs, but only rewards, which introduces a bias that is observable by users. Between subjects, we vary the accuracy and bias of the DSS, and observe the participants' total score, time to completion, the extent to which they follow or ignore suggestions. We also measure the acceptability of the DSS in an exit survey. Our results show that participants tend to score better with the DSS, that the score increase is due to users following the DSS advice, and related to the difficulty of the game and the accuracy of the DSS. We observe that this setting elicits mostly rational behavior from participants, who place a moderate amount of trust in the DSS and show neither algorithmic aversion (under-reliance) nor automation bias (over-reliance).However, their stated willingness to accept the DSS in the exit survey seems less sensitive to the accuracy of the DSS than their behavior, suggesting that users are only partially aware of the (lack of) accuracy of the DSS.

Performance Evaluation Of The Novel Dss Based On Humanware Service

A Framework of Service Invocation in the Novel DSS Based on Humanware Service

RESEARCH ON NOVEL DECISION SYSTEM KEY TECHNOLOGIES BASED ON HUMANWARE SERVICE

Effectiveness Evaluation of Humanware Service in Novel Decision System.

A Conceptual Framework of Service Invocation in the Novel DSS: the Example in Situation Assessment

Development of a Service-Oriented Sharable Clinical Decision Support System Based on Ontology for Chronic Disease.

Humanware Model in Novel Decision System

Research on Interface of Humanware Service-Oriented Decision Support System??

A Brief Review on Models for Performance Evaluation in DSS Architecture

Semiparallel Service: Theory and Application

Data Mining Service Architecture - an Exploration on Self-organizing Software.

Human Response to an AI-Based Decision Support System: A User Study on the Effects of Accuracy and Bias

AIDSS-HR: An Automated Intelligent Decision Support System for Enhancing the Performance of Employees

Real-time decision support for human-machine interaction in digital railway control rooms

A Performance Evaluation Method and It's Implementation for Web Service

Dynamic Scalable Stochastic Petri Net: A Novel Model for Designing and Analysis of Resource Scheduling in Cloud Computing

Analysis of Intelligent Decision Support Systems and a Multi Criteria Framework for Assessment

Simulative Evaluation on Human Machine Interface in DCS Main Control Room of Power Plant

Applications and Implementation DECISION SUPPORT SYSTEMS: AN EMPIRICAL ASSESSMENT FOR DECISION MAKING

Service-Oriented Enterprise Network Performance Analysis

Decision Support System of Employee Performance Evaluation