Abstract:Introduction: In the past year, the use of large language models (LLMs) has generated significant interest and excitement because of their potential to revolutionise various fields, including medical education for aspiring physicians. Although medical students undergo a demanding educational process to become competent health care professionals, the emergence of LLMs presents a promising solution to challenges like information overload, time constraints and pressure on clinical educators. However, integrating LLMs into medical education raises critical concerns and challenges for educators, professionals and students. This systematic review aims to explore LLM applications in medical education, specifically their impact on medical students' learning experiences. Methods: A systematic search was performed in PubMed, Web of Science and Embase for articles discussing the applications of LLMs in medical education using selected keywords related to LLMs and medical education, from the time of ChatGPT's debut until February 2024. Only articles available in full text or English were reviewed. The credibility of each study was critically appraised by two independent reviewers. Results: The systematic review identified 166 studies, of which 40 were found by review to be relevant to the study. Among the 40 relevant studies, key themes included LLM capabilities, benefits such as personalised learning and challenges regarding content accuracy. Importantly, 42.5% of these studies specifically evaluated LLMs in a novel way, including ChatGPT, in contexts such as medical exams and clinical/biomedical information, highlighting their potential in replicating human-level performance in medical knowledge. The remaining studies broadly discussed the prospective role of LLMs in medical education, reflecting a keen interest in their future potential despite current constraints. Conclusions: The responsible implementation of LLMs in medical education offers a promising opportunity to enhance learning experiences. However, ensuring information accuracy, emphasising skill-building and maintaining ethical safeguards are crucial. Continuous critical evaluation and interdisciplinary collaboration are essential for the appropriate integration of LLMs in medical education.

LLM-based automatic short answer grading in undergraduate medical education

Towards LLM-based Autograding for Short Textual Answers

The Potential of LLMs in Medical Education: Generating Questions and Answers for Qualification Exams

Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset

Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis

Large language models for generating medical examinations: systematic review

Grade Like a Human: Rethinking Automated Assessment with Large Language Models

Towards Leveraging Large Language Models for Automated Medical Q&A Evaluation

Automatic Short Answer Grading via Multiway Attention Networks

Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Model

A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization

ASAG2024: A Combined Benchmark for Short Answer Grading

Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring

"I understand why I got this grade": Automatic Short Answer Grading with Feedback

Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education

A Benchmark for Long-Form Medical Question Answering

Performance of the pre-trained large language model GPT-4 on automated short answer grading

AI-assisted Automated Short Answer Grading of Handwritten University Level Mathematics Exams

Large Language Models As MOOCs Graders

A systematic review of large language models and their implications in medical education

The Role of Large Language Models in Medical Education: Applications and Implications