Automated abstraction of clinical parameters of multiple myeloma from real-world clinical notes using large language models

Alana O'Brien Del Campo,Dmytro Lituiev,Gowtham Varma,Mithun Manoharan,Sunil Kumar Ravi,Avinash Aman,Ankit Kansagra,Joel Greshock,AJ Venkatakrishnan,Ashita Batavia
DOI: https://doi.org/10.1101/2024.12.17.24318605
2024-12-20
Abstract:Background: Real–world evidence (RWE) is increasingly recognized as a valuable type of oncology research but extracting fit–for–purpose real–world data (RWD) from electronic health records (EHRs) remains challenging. Manual abstraction from free–text clinical documents, although the gold standard for information extraction, is resource–intensive. RWD generation using natural language processing (NLP) has been limited by performance ceilings and annotation requirements, which recent LLMs improve on. We evaluate new NLP workflows in abstracting multiple myeloma (MM) information from de–identified EHRs. Methods: NLP workflows (BERT and Llama–based using various prompt types) were developed for 12 MM–specific data fields and evaluated with manually curated data from 125 clinical notes. The best Llama–based workflow for three data fields was applied to all recent notes in selected charts to generate patient journey timelines. Results: Average F1 for the best Llama and BERT workflows was 0.82 and 0.65 respectively. Best workflow performance ranged across the data fields (F1 = 0.59—0.99). Statistical analysis of the results showed model size, inter–rater reliability (IRR), variable type, and prompt design significantly predicted workflow performance, in descending order of significance (p < 0.05). Conclusion: The overall performance improvements seen with larger LLMs and chain–of–thought prompting was greater in ambiguous data fields. IRR can be used to prioritize NLP resources and increase efficiency of RWD generation without sacrificing data quality.
What problem does this paper attempt to address?