Abstract:As the health care industry increasingly embraces large language models (LLMs), understanding the consequence of this integration becomes crucial for maximizing benefits while mitigating potential pitfalls. This paper explores the evolving relationship among clinician trust in LLMs, the transition of data sources from predominantly human-generated to artificial intelligence (AI)-generated content, and the subsequent impact on the performance of LLMs and clinician competence. One of the primary concerns identified in this paper is the LLMs' self-referential learning loops, where AI-generated content feeds into the learning algorithms, threatening the diversity of the data pool, potentially entrenching biases, and reducing the efficacy of LLMs. While theoretical at this stage, this feedback loop poses a significant challenge as the integration of LLMs in health care deepens, emphasizing the need for proactive dialogue and strategic measures to ensure the safe and effective use of LLM technology. Another key takeaway from our investigation is the role of user expertise and the necessity for a discerning approach to trusting and validating LLM outputs. The paper highlights how expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining a critical oversight to identify and correct potential inaccuracies in AI-generated content. This balance of trust and skepticism is vital for ensuring that LLMs augment rather than undermine the quality of patient care. We also discuss the risks associated with the deskilling of health care professionals. Frequent reliance on LLMs for critical tasks could result in a decline in health care providers' diagnostic and thinking skills, particularly affecting the training and development of future professionals. The legal and ethical considerations surrounding the deployment of LLMs in health care are also examined. We discuss the medicolegal challenges, including liability in cases of erroneous diagnoses or treatment advice generated by LLMs. The paper references recent legislative efforts, such as The Algorithmic Accountability Act of 2023, as crucial steps toward establishing a framework for the ethical and responsible use of AI-based technologies in health care. In conclusion, this paper advocates for a strategic approach to integrating LLMs into health care. By emphasizing the importance of maintaining clinician expertise, fostering critical engagement with LLM outputs, and navigating the legal and ethical landscape, we can ensure that LLMs serve as valuable tools in enhancing patient care and supporting health care professionals. This approach addresses the immediate challenges posed by integrating LLMs and sets a foundation for their maintainable and responsible use in the future.

Clinician voices on ethics of LLM integration in healthcare: a thematic analysis of ethical concerns and implications

Ethical and regulatory challenges of large language models in medicine

The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)

Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Health Care Professionals

Ethical Considerations and Fundamental Principles of Large Language Models in Medical Education: Viewpoint

Empathy and Equity: Key Considerations for Large Language Model Adoption in Health Care

Navigating LLM Ethics: Advancements, Challenges, and Future Directions

Large language models in medical ethics: useful but not expert

Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Healthcare Professionals

Embedded values-like shape ethical reasoning of large language models on primary care ethical dilemmas

Bias patterns in the application of LLMs for clinical decision support: A comprehensive study

A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics

"The teachers are confused as well": A Multiple-Stakeholder Ethics Discussion on Large Language Models in Computing Education

Leveraging large language models to foster equity in healthcare

Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics

"I'm categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices

LLMs as mirrors of societal moral standards: reflection of cultural divergence and agreement across ethical topics

Discussing the Role of Explainable AI and Evaluation Frameworks for Safe and Effective Integration of Large Language Models in Healthcare

Revolutionizing Healthcare: the Transformative Impact of LLMs in Medicine (Preprint)