Abstract:Objectives: The integration of these preventive guidelines with Electronic Health Records (EHRs) systems, coupled with the generation of personalized preventive care recommendations, holds significant potential for improving healthcare outcomes. Our study investigates the feasibility of using Large Language Models (LLMs) to automate the assessment criteria and risk factors from the guidelines for future analysis against medical records in EHR. Materials and methods: We annotated the criteria, risk factors, and preventive medical services described in the adult guidelines published by United States Preventive Services Taskforce and evaluated 3 state-of-the-art LLMs on extracting information in these categories from the guidelines automatically. Results: We included 24 guidelines in this study. The LLMs can automate the extraction of all criteria, risk factors, and medical services from 9 guidelines. All 3 LLMs perform well on extracting information regarding the demographic criteria or risk factors. Some LLMs perform better on extracting the social determinants of health, family history, and preventive counseling services than the others. Discussion: While LLMs demonstrate the capability to handle lengthy preventive care guidelines, several challenges persist, including constraints related to the maximum length of input tokens and the tendency to generate content rather than adhering strictly to the original input. Moreover, the utilization of LLMs in real-world clinical settings necessitates careful ethical consideration. It is imperative that healthcare professionals meticulously validate the extracted information to mitigate biases, ensure completeness, and maintain accuracy. Conclusion: We developed a data structure to store the annotated preventive guidelines and make it publicly available. Employing state-of-the-art LLMs to extract preventive care criteria, risk factors, and preventive care services paves the way for the future integration of these guidelines into the EHR.

Considerations for health care institutions training large language models on electronic health records

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Is larger always better? Evaluating and prompting large language models for non-generative medical tasks

Large language models in health care: Development, applications, and challenges

Harnessing the Power of Large Language Models (LLMs) for Electronic Health Records (EHRs) Optimization

A Study of Generative Large Language Model for Medical Research and Healthcare

A Survey of Clinicians’ Views of the Utility of Large Language Models

A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations

Large language models in medicine: the potentials and pitfalls

A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics

A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

Evaluating large language models in medical applications: a survey

Large Language Models in Healthcare: A Comprehensive Benchmark

Ethical and regulatory challenges of large language models in medicine

Zero-shot learning to extract assessment criteria and medical services from the preventive healthcare guidelines using large language models

Large Language Model Benchmarks in Medical Tasks

The future landscape of large language models in medicine