The doctor will polygraph you now: ethical concerns with AI for fact-checking patients

James Anibal,Jasmine Gunkel,Hannah Huth,Hang Nguyen,Shaheen Awan,Yael Bensoussan,Bradford Wood

2024-08-15

Abstract:Clinical artificial intelligence (AI) methods have been proposed for predicting social behaviors which could be reasonably understood from patient-reported data. This raises ethical concerns about respect, privacy, and patient awareness/control over how their health data is used. Ethical concerns surrounding clinical AI systems for social behavior verification were divided into three main categories: (1) the use of patient data retrospectively without informed consent for the specific task of verification, (2) the potential for inaccuracies or biases within such systems, and (3) the impact on trust in patient-provider relationships with the introduction of automated AI systems for fact-checking. Additionally, this report showed the simulated misuse of a verification system and identified a potential LLM bias against patient-reported information in favor of multimodal data, published literature, and the outputs of other AI methods (i.e., AI self-trust). Finally, recommendations were presented for mitigating the risk that AI verification systems will cause harm to patients or undermine the purpose of the healthcare system.

Computers and Society

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is: **ethical issues arising from the use of artificial intelligence (AI) systems to verify patients' social behaviors**. Specifically, the paper explores the following ethical challenges:

1. **Retrospective data use without explicit patient consent**:
- Using electronic health records (EHR) or insurance data to verify patients' self - reported information may be carried out without obtaining the patient's informed consent for a specific task, which violates the patient's right to privacy and data control.

2. **Inaccuracy and bias of algorithms**:
- Predictions made by AI systems may be biased, especially when they perform inconsistently among different populations (such as race, gender, socioeconomic status, etc.), which may lead to unfair treatment of certain groups and further exacerbate systemic biases in the medical system.

3. **Impact on the trust in the doctor - patient relationship**:
- The introduction of automated AI systems for "fact - checking" may weaken the trust relationship between patients and doctors. When an AI system questions a patient's self - report, the patient may feel disrespected and even doubt their honesty, thus affecting the quality of doctor - patient communication.

In addition, the paper also demonstrates the possibility of simulating the abuse of verification systems and points out that large - language models (LLM) may be biased and more likely to trust multi - modal data, published literature, and the outputs of other AI methods rather than the information provided by patients (i.e., "AI self - confidence"). This tendency may exacerbate the above - mentioned ethical issues.

To address these risks, the paper makes several suggestions, including strictly defining the usage scenarios of clinical AI systems, ensuring patient informed consent, maintaining the participation of human experts in the decision - making process, and formulating relevant policies to prevent technological abuse.