Abstract:As the health care industry increasingly embraces large language models (LLMs), understanding the consequence of this integration becomes crucial for maximizing benefits while mitigating potential pitfalls. This paper explores the evolving relationship among clinician trust in LLMs, the transition of data sources from predominantly human-generated to artificial intelligence (AI)-generated content, and the subsequent impact on the performance of LLMs and clinician competence. One of the primary concerns identified in this paper is the LLMs' self-referential learning loops, where AI-generated content feeds into the learning algorithms, threatening the diversity of the data pool, potentially entrenching biases, and reducing the efficacy of LLMs. While theoretical at this stage, this feedback loop poses a significant challenge as the integration of LLMs in health care deepens, emphasizing the need for proactive dialogue and strategic measures to ensure the safe and effective use of LLM technology. Another key takeaway from our investigation is the role of user expertise and the necessity for a discerning approach to trusting and validating LLM outputs. The paper highlights how expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining a critical oversight to identify and correct potential inaccuracies in AI-generated content. This balance of trust and skepticism is vital for ensuring that LLMs augment rather than undermine the quality of patient care. We also discuss the risks associated with the deskilling of health care professionals. Frequent reliance on LLMs for critical tasks could result in a decline in health care providers' diagnostic and thinking skills, particularly affecting the training and development of future professionals. The legal and ethical considerations surrounding the deployment of LLMs in health care are also examined. We discuss the medicolegal challenges, including liability in cases of erroneous diagnoses or treatment advice generated by LLMs. The paper references recent legislative efforts, such as The Algorithmic Accountability Act of 2023, as crucial steps toward establishing a framework for the ethical and responsible use of AI-based technologies in health care. In conclusion, this paper advocates for a strategic approach to integrating LLMs into health care. By emphasizing the importance of maintaining clinician expertise, fostering critical engagement with LLM outputs, and navigating the legal and ethical landscape, we can ensure that LLMs serve as valuable tools in enhancing patient care and supporting health care professionals. This approach addresses the immediate challenges posed by integrating LLMs and sets a foundation for their maintainable and responsible use in the future.

Key Insights for the Ethical and Appropriate Use of Artificial Intelligence by Medical Learners

Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine

Ethical Considerations and Fundamental Principles of Large Language Models in Medical Education: Viewpoint

Generative artificial intelligence (AI) in higher education: a comprehensive review of challenges, opportunities, and implications

Adapting to the Impact of AI in Scientific Writing: Balancing Benefits and Drawbacks while Developing Policies and Regulations

Generative AI and medical ethics: the state of play

The transformative impact of large language models on medical writing and publishing: current applications, challenges and future directions

Constructing knowledge: the role of AI in medical learning

Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Health Care Professionals

Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions

The global landscape of academic guidelines for generative AI and Large Language Models

Guiding principles and proposed classification system for the responsible adoption of artificial intelligence in scientific writing in medicine

The Role of Large Language Models in Medical Education: Applications and Implications

A systematic review of large language models and their implications in medical education

A review of ophthalmology education in the era of generative artificial intelligence

Large Language Models in Ophthalmology: Potential and Pitfalls

Artificial intelligence and pain medicine education: Benefits and pitfalls for the medical trainee

Harnessing the potential of large language models in medical education: promise and pitfalls

Impact of ChatGPT and Large Language Models on Radiology Education: Association of Academic Radiology—Radiology Research Alliance Task Force White Paper

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

Large language models in medical ethics: useful but not expert