Abstract:High-stakes prediction tasks (e.g., patient diagnosis) are often handled by trained human experts. A common source of concern about automation in these settings is that experts may exercise intuition that is difficult to model and/or have access to information (e.g., conversations with a patient) that is simply unavailable to a would-be algorithm. This raises a natural question whether human experts add value which could not be captured by an algorithmic predictor. We develop a statistical framework under which we can pose this question as a natural hypothesis test. Indeed, as our framework highlights, detecting human expertise is more subtle than simply comparing the accuracy of expert predictions to those made by a particular learning algorithm. Instead, we propose a simple procedure which tests whether expert predictions are statistically independent from the outcomes of interest after conditioning on the available inputs (`features'). A rejection of our test thus suggests that human experts may add value to any algorithm trained on the available data, and has direct implications for whether human-AI `complementarity' is achievable in a given prediction task. We highlight the utility of our procedure using admissions data collected from the emergency department of a large academic hospital system, where we show that physicians' admit/discharge decisions for patients with acute gastrointestinal bleeding (AGIB) appear to be incorporating information that is not available to a standard algorithmic screening tool. This is despite the fact that the screening tool is arguably more accurate than physicians' discretionary decisions, highlighting that -- even absent normative concerns about accountability or interpretability -- accuracy is insufficient to justify algorithmic automation.

Man Versus Machine: Complex Estimates and Auditor Reliance on Artificial Intelligence

How do financial executives respond to the use of artificial intelligence in financial reporting and auditing?

Artificial Intelligence in Auditing: A Conceptual Framework for Auditing Practices

The Ethical Implications of Using Artificial Intelligence in Auditing

Is Artificial Intelligence Making Audit Firms More Efficient?

Is artificial intelligence improving the audit process?

An analysis of auditors’ perceptions towards artificial intelligence and its contribution to audit quality

Effects of Artificial Intelligence in the Modern Business Client Artificial Intelligence Application and Audit Quality

How Does Artificial Intelligence Shape Audit Firms?

Adoption of artificial intelligence in auditing: An exploratory study

Auditing for Human Expertise

Perception Of Artificial Intelligence Adoption on Audit Quality

Navigating the Manager-AI Divide

Artificial intelligence and audit quality: Implications for practicing accountants

How Does Artificial Intelligence Shape the Audit Industry?

Watch Me Improve—Algorithm Aversion and Demonstrating the Ability to Learn

Does the adoption of artificial intelligence by audit firms and their clients affect audit quality and efficiency? Evidence from China

In What Way Does Artificial Intelligence Influences Audit Practice? Empirical Evidence from Southwest, Nigeria

Algorithm aversion, emotions, and investor reaction: Does disclosing the use of AI influence investment decisions?