Abstract:BACKGROUND. Medical decision-making is commonly guided by systematic analysis of peer-reviewed scientific literature, published as systematic literature reviews (SLRs). These analyses are cumbersome to conduct as they require large amounts of time and subject matter expertise to be available. Automated extraction of key datapoints from clinical publications could speed up the process of systematic literature review assembly. To this end, we built, trained and validated SURUS, a named entity recognition (NER) system comprised of a Bidirectional Encoder Representations from Transformers (BERT) model trained on a highly granular dataset. The aim of this study was to assess the quality of classification of critical elements in clinical study abstracts by SURUS, in particular the patient, intervention, comparator and outcome (PICO) elements and elements of study design. DATASET & METHODS. The PubMedBERT-based model was trained and evaluated using a dataset of 400 interventional study abstracts, manually annotated by experts using 25 labels with a total of 39,531 annotations according to a strict annotation guideline, with Cohen's kappa inter-annotator agreement of 0.81. We evaluated in-domain quality, and assessed out-of-domain quality of the system by testing it on out-of-domain abstracts of other disease areas and observational study types. Finally, we tested the utility of SURUS by comparing its predictions to expert-assigned PICO and study design (PICOS) classifications. RESULTS. The SURUS NER system achieved an overall F1 score of 0.95, with minor deviation between labels. In addition, SURUS achieved a NER F1 of 0.90 for out-of-domain therapeutic area abstracts and 0.84 for observational study abstracts. Finally, SURUS showed considerable utility when compared to expert-assigned PICOS classifications of interventional studies, with an F1 of 0.89 and a recall of 0.96. CONCLUSION. To our knowledge, with an F1 score of 0.95, SURUS ranks among the best-performing models available to date for conducting exhaustive systematic literature analyses. A strict guideline and high inter-annotation agreement resulted in high-quality in-domain medical entity of a finetuned BERT-based model, which was largely preserved during extensive out-of-domain evaluation, indicating its utility across other indication areas and study types. Current approaches in the field lack the granularity in training data and versatility demonstrated by the SURUS approach, thereby making the latter the preferred choice for automated extraction and classification tasks in the clinical literature domain. We think that this approach sets a new standard in medical literature analysis and paves the way for creating highly granular datasets of labelled entities that can be used for downstream analysis outside of traditional SLRs.

Evaluation of SURUS: a Named Entity Recognition System to Extract Knowledge from Interventional Study Records

Evaluation of a prototype machine learning tool to semi-automate data extraction for systematic literature reviews

Extracting Systemic Anticancer Therapy and Response Information From Clinical Notes Following the RECIST Definition

Automated medical chart review for breast cancer outcomes research: a novel natural language processing extraction system

AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review

Granularly, Precisely, and Timely: Leveraging Large Language Models for Safety and Efficacy Extraction in Oncology Clinical Trial Abstracts (SEETrials)

SEETrials: Leveraging Large Language Models for Safety and Efficacy Extraction in Oncology Clinical Trials

Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature

An annotated corpus of clinical trial publications supporting schema-based relational information extraction

SNOBERT: A Benchmark for clinical notes entity linking in the SNOMED CT clinical terminology

A Novel Framework to Expedite Systematic Reviews by Automatically Building Information Extraction Training Corpora

Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline

AI-assisted Knowledge Discovery in Biomedical Literature to Support Decision-making in Precision Oncology

Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations

Cross-institution natural language processing for reliable clinical association studies: a methodological exploration

Clinical Decision Support and Natural Language Processing in Medicine: Systematic Literature Review

CCS Explorer: Relevance Prediction, Extractive Summarization, and Named Entity Recognition from Clinical Cohort Studies

A Cross-institutional Evaluation on Breast Cancer Phenotyping NLP Algorithms on Electronic Health Records

Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations

Natural Language Processing Model for Identifying Critical Findings-A Multi-Institutional Study

An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts