Abstract:Evidence-based medicine (EBM) represents a paradigm of providing patient care grounded in the most current and rigorously evaluated research. Recent advances in large language models (LLMs) offer a potential solution to transform EBM by automating labor-intensive tasks and thereby improving the efficiency of clinical decision-making. This study explores integrating LLMs into the key stages in EBM, evaluating their ability across evidence retrieval (PICO extraction, biomedical question answering), synthesis (summarizing randomized controlled trials), and dissemination (medical text simplification). We conducted a comparative analysis of seven LLMs, including both proprietary and open-source models, as well as those fine-tuned on medical corpora. Specifically, we benchmarked the performance of various LLMs on each EBM task under zero-shot settings as baselines, and employed prompting techniques, including in-context learning, chain-of-thought reasoning, and knowledge-guided prompting to enhance their capabilities. Our extensive experiments revealed the strengths of LLMs, such as remarkable understanding capabilities even in zero-shot settings, strong summarization skills, and effective knowledge transfer via prompting. Promoting strategies such as knowledge-guided prompting proved highly effective (e.g., improving the performance of GPT-4 by 13.10% over zero-shot in PICO extraction). However, the experiments also showed limitations, with LLM performance falling well below state-of-the-art baselines like PubMedBERT in handling named entity recognition tasks. Moreover, human evaluation revealed persisting challenges with factual inconsistencies and domain inaccuracies, underscoring the need for rigorous quality control before clinical application. This study provides insights into enhancing EBM using LLMs while highlighting critical areas for further research. The code is publicly available on Github.

OptimalMEE: Optimizing Large Language Models for Medical Event Extraction Through Fine-Tuning and Post-hoc Verification

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Can LLMs' Tuning Methods Work in Medical Multimodal Domain?

Efficient Fine-Tuning of Large Language Models for Automated Medical Documentation

Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding

PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models

Benchmarking Large Language Models in Evidence-Based Medicine

Towards Evaluating and Building Versatile Large Language Models for Medicine

Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models

MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts

PMC-LLaMA: Further Finetuning LLaMA on Medical Papers

MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

Large Language Model Benchmarks in Medical Tasks

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

LLMs in Biomedicine: A study on clinical Named Entity Recognition

Evaluating large language models in medical applications: a survey

A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations