Developing a natural language processing system using transformer-based models for adverse drug event detection in electronic health records

Jingyuan Wu,Xiaodi Ruan,Elizabeth McNeer,Katelyn M. Rossow,Leena Choi
DOI: https://doi.org/10.1101/2024.07.09.24310100
2024-07-10
Abstract:Objective: To develop a transformer-based natural language processing (NLP) system for detecting adverse drug events (ADEs) from clinical notes in electronic health records (EHRs). Materials and Methods: We fine-tuned BERT Short-Formers and Clinical-Longformer using the processed dataset of the 2018 National NLP Clinical Challenges (n2c2) shared task Track 2. We investigated two data processing methods, window-based and split-based approaches, to find an optimal processing method. We evaluated the generalization capabilities on a dataset extracted from Vanderbilt University Medical Center (VUMC) EHRs. Results: On the n2c2 dataset, the best average macro F-scores of 0.832 and 0.868 were achieved using a 15-word window with PubMedBERT and a 10-chunk split with Clinical-Longformer. On the VUMC dataset, the best average macro F-scores of 0.720 and 0.786 were achieved using a 4-chunk split with PubMedBERT and Clinical-Longformer. Discussion: Our study provided a comparative analysis of data processing methods. The fine-tuned transformer models showed good performance for ADE-related tasks. Especially, Clinical-Longformer model with split-based approach had a great potential for practical implementation of ADE detection. While the token limit was crucial, the chunk size also significantly influenced model performance, even when the text length was within the token limit. Conclusion: We provided guidance on model development, including data processing methods for ADE detection from clinical notes using transformer-based models. Our results on two datasets indicated that data processing methods and models should be carefully selected based on the type of clinical notes and the allocation trade-offs of human and computational power in annotation and model fine-tuning.
Health Informatics
What problem does this paper attempt to address?
The paper aims to develop a natural language processing (NLP) system based on the Transformer model to detect adverse drug events (ADEs) from clinical notes in electronic health records (EHRs). The researchers fine-tuned BERT Short-Formers (including various variants) and Clinical-Longformer, and compared window-based and chunk-based data processing methods to determine the best strategy. The evaluation was conducted on the 2018 National NLP Clinical Challenge (n2c2) dataset and the Vanderbilt University Medical Center (VUMC) EHR dataset. The paper found that using a window size of 15 tokens for PubMedBERT and 10 segments for Clinical-Longformer achieved the best average macro F-score on the n2c2 dataset. On the VUMC dataset, the best results were obtained from 4 segments of PubMedBERT and Clinical-Longformer respectively. The research emphasized the importance of data processing methods and model selection, as well as the impact of text length, token limitations, and context completeness on model performance. The paper also pointed out that although the BERT model and its variants have performed well in ADE-related tasks, they have limitations in processing long texts. Clinical-Longformer addresses this issue with its innovative attention mechanism, enabling it to handle longer sequences while maintaining high performance. Overall, this work provides guidance for extracting ADE information from clinical notes using the Transformer model, which helps in early identification and prevention of adverse drug events, thereby improving patient care and reducing costs.