Abstract:Results from Randomized Controlled Trials (RCTs) establish the comparative effectiveness of interventions, and are in turn critical inputs for evidence-based care. However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles. This onerous manual process has motivated work on (semi-)automating extraction of structured evidence from trial reports. In this work we propose and evaluate a text-to-text model built on instruction-tuned Large Language Models (LLMs) to jointly extract Interventions, Outcomes, and Comparators (ICO elements) from clinical abstracts, and infer the associated results reported. Manual (expert) and automated evaluations indicate that framing evidence extraction as a conditional generation task and fine-tuning LLMs for this purpose realizes considerable ($\sim$20 point absolute F1 score) gains over the previous SOTA. We perform ablations and error analyses to assess aspects that contribute to model performance, and to highlight potential directions for further improvements. We apply our model to a collection of published RCTs through mid-2022, and release a searchable database of structured findings: <a class="link-external link-http" href="http://ico-relations.ebm-nlp.com" rel="external noopener nofollow">this http URL</a>

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenge of automatically extracting interventions, outcomes, and research findings from randomized controlled trial (RCT) reports. Specifically, the results of RCTs are crucial for evidence - based medical practice, but these results usually appear in the form of unstructured natural language in articles describing trial design, implementation, and results. Clinicians need to manually extract findings related to specific interventions and outcomes from these articles, which is a time - consuming and labor - intensive process. Therefore, this study aims to develop a text - to - text model capable of jointly extracting interventions, outcomes, and comparators (referred to as ICO elements) and inferring the relevant results reported by leveraging the capabilities of large - language models (LLMs). This method not only improves the extraction efficiency but also significantly enhances the extraction accuracy, with the F1 score increasing by approximately 20 percentage points compared to previous methods. The key contributions of the paper include: - **Method innovation**: Proposed an instruction - tuned large - language model to perform conditional generation tasks for jointly extracting ICO elements and related results from clinical abstracts. - **Performance improvement**: Achieved a significant performance improvement relative to the previous best method through fine - tuning of the model. - **Evaluation and analysis**: Conducted exhaustive manual and automated evaluations and explored factors affecting model performance and directions for future improvement through ablation experiments and error analysis. - **Application and resources**: Applied the model to the RCT collection published as of mid - 2022 and released a searchable database of structured findings for use by researchers and clinicians. In summary, this paper solves the problem of efficiently and accurately extracting key information from RCT reports, which is of great significance for promoting evidence - based medical practice.

Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs

Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models

Inferring Which Medical Treatments Work from Reports of Clinical Trials

Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study

How good are large language models for automated data extraction from randomized trials?

Benchmarking Large Language Models in Evidence-Based Medicine

Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations

Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations

Performance of two large language models for data extraction in evidence synthesis

NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports

Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews

Automated Extraction of Patient-Centered Outcomes After Breast Cancer Treatment: An Open-Source Large Language Model-Based Toolkit

Learning to match patients to clinical trials using large language models

Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study

AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs

Using large language models for safety-related table summarization in clinical study reports

Automated Clinical Data Extraction with Knowledge Conditioned LLMs

Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges

From RAGs to riches: Using large language models to write documents for clinical trials

A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation