ChatGPT-o1 and the Pitfalls of Familiar Reasoning in Medical Ethics

Shelly Soffer,Vera Sorin,Girish Nadkarni,Eyal Klang

DOI: https://doi.org/10.1101/2024.09.25.24314342

2024-09-27

Abstract:Large language models (LLMs) like ChatGPT often exhibit Type 1 thinking - fast, intuitive reasoning that relies on familiar patterns-which can be dangerously simplistic in complex medical or ethical scenarios requiring more deliberate analysis. In our recent explorations, we observed that LLMs frequently default to well-known answers, failing to recognize nuances or twists in presented situations. For instance, when faced with modified versions of the classic "Surgeon's Dilemma" or medical ethics cases where typical dilemmas were resolved, LLMs still reverted to standard responses, overlooking critical details. Even models designed for enhanced analytical reasoning, such as ChatGPT-o1, did not consistently overcome these limitations. This suggests that despite advancements toward fostering Type 2 thinking, LLMs remain heavily influenced by familiar patterns ingrained during training. As LLMs are increasingly integrated into clinical practice, it is crucial to acknowledge and address these shortcomings to ensure reliable and contextually appropriate AI assistance in medical decision-making.

What problem does this paper attempt to address?

The paper attempts to address the limitations of large language models (LLMs) in handling complex medical ethical scenarios. Specifically, these models tend to exhibit "Type 1 thinking," which is a fast, intuitive reasoning style that relies on familiar patterns and experiences. However, in medical and ethical contexts that require detailed analysis and consideration of complex information (i.e., "Type 2 thinking"), this quick reasoning approach can lead to dangerously oversimplified conclusions.

Researchers found through a series of tests that even models improved to enhance analytical capabilities (such as ChatGPT-o1) still default to familiar but not entirely appropriate answers for the current context. For example, in a modified "surgeon's dilemma" case, despite clear information indicating that the father is a surgeon and the mother is a social worker, and only the boy had an accident, the model still arrived at an incorrect conclusion. Similarly, in some medical ethics cases, even after resolving typical dilemmas, the model continued to discuss standard ethical debates.

Therefore, the paper emphasizes the need to recognize these limitations and make further improvements and refinements before integrating LLMs into clinical practice to ensure the reliability and appropriateness of AI assistance in medical decision-making.