Abstract:Introduction Plain language summaries (PLSs) make complex healthcare evidence accessible to patients and the public. Large language models (LLMs) may assist in generating accurate, readable PLSs. This study explored using the LLM Claude 2 to create PLSs of evidence reviews from the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program. Methods We selected 10 evidence reviews published from 2021 to 2023, representing a range of methods and topics. We iteratively developed a prompt to guide Claude 2 in creating PLSs which included specifications for plain language, reading level, length, organizational structure, active voice, and inclusive language. PLSs were assessed for adherence to prompt specifications, comprehensiveness, accuracy, readability, and cultural sensitivity. Results All PLSs met the word count. We judged one PLS as fully comprehensive; seven mostly comprehensive. We judged two PLSs as fully capturing the PICO elements; five with minor PICO errors. We judged three PLSs as accurately reporting the results; and four with minor result errors. We judged three PLSs as having major result errors for incorrectly reporting total participants. Five PLSs met the target 6th to 8th grade reading level. Passive voice use averaged 16%. All PLSs used inclusive language. Conclusions LLMs show promise for assisting in PLS creation but likely require human input to ensure accuracy, comprehensiveness, and the appropriate nuances of interpretation. Iterative prompt refinement may improve results and address the needs of specific reviews and audiences. As text‐only summaries, the AI‐generated PLSs could not meet all consumer communication criteria, such as textual design and visual representations. Further testing should include consumer reviewers and explore how to best leverage LLM support in drafting PLS text for complex evidence reviews.

Using large language models for safety-related table summarization in clinical study reports

Using Large Language Models to Generate Clinical Trial Tables and Figures

From RAGs to riches: Using large language models to write documents for clinical trials

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

[Clinical application of large language models : Does ChatGPT replace medical report formulation? An experience report]

Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Synoptic Reporting by Summarizing Cancer Pathology Reports using Large Language Models

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patients

A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation

Automatic Summarization of Doctor-Patient Encounter Dialogues Using Large Language Model through Prompt Tuning

The use of a large language model to create plain language summaries of evidence reviews in healthcare: A feasibility study

The use of large language models to enhance cancer clinical trial educational materials

Transforming clinical trials: the emerging roles of large language models

A Study of Generative Large Language Model for Medical Research and Healthcare

A Survey of Clinicians’ Views of the Utility of Large Language Models

Large language models encode clinical knowledge

The long but necessary road to responsible use of large language models in healthcare research