Abstract:Background: Negation and speculation unrelated to abnormal findings can lead to false-positive alarms for automatic radiology report highlighting or flagging by laboratory information systems. Objective: This internal validation study evaluated the performance of natural language processing methods (NegEx, NegBio, NegBERT, and transformers). Methods: We annotated all negative and speculative statements unrelated to abnormal findings in reports. In experiment 1, we fine-tuned several transformer models (ALBERT [A Lite Bidirectional Encoder Representations from Transformers], BERT [Bidirectional Encoder Representations from Transformers], DeBERTa [Decoding-Enhanced BERT With Disentangled Attention], DistilBERT [Distilled version of BERT], ELECTRA [Efficiently Learning an Encoder That Classifies Token Replacements Accurately], ERNIE [Enhanced Representation through Knowledge Integration], RoBERTa [Robustly Optimized BERT Pretraining Approach], SpanBERT, and XLNet) and compared their performance using precision, recall, accuracy, and F1-scores. In experiment 2, we compared the best model from experiment 1 with 3 established negation and speculation-detection algorithms (NegEx, NegBio, and NegBERT). Results: Our study collected 6000 radiology reports from 3 branches of the Chi Mei Hospital, covering multiple imaging modalities and body parts. A total of 15.01% (105,755/704,512) of words and 39.45% (4529/11,480) of important diagnostic keywords occurred in negative or speculative statements unrelated to abnormal findings. In experiment 1, all models achieved an accuracy of >0.98 and F1-score of >0.90 on the test data set. ALBERT exhibited the best performance (accuracy=0.991; F1-score=0.958). In experiment 2, ALBERT outperformed the optimized NegEx, NegBio, and NegBERT methods in terms of overall performance (accuracy=0.996; F1-score=0.991), in the prediction of whether diagnostic keywords occur in speculative statements unrelated to abnormal findings, and in the improvement of the performance of keyword extraction (accuracy=0.996; F1-score=0.997). Conclusions: The ALBERT deep learning method showed the best performance. Our results represent a significant advancement in the clinical applications of computer-aided notification systems.

Term Extraction and Negation Detection Method in Chinese Clinical Document

A finite-state automata based negation detection algorithm for Chinese clinical documents

Lexical Characteristics Analysis of Chinese Clinical Documents

Detecting Negation and Scope in Chinese Clinical Notes Using Character and Word Embedding

Application of Chinese medical document anonymization in EMR system

Temporal Expression Recognition and Temporal Relationship Extraction from Chinese Narrative Medical Records

Identification of Adverse Drug Events in Chinese Clinical Narrative Text

A CRF-based Method for Automatic Construction of Chinese Symptom Lexicon

The Automatic Extraction of Temporal Relation from Chinese Narrative Medical Records Using Conditional Random Fields

Traditional Chinese Medicine Clinical Records Classification Using Knowledge-Powered Document Embedding

A Novel Approach Towards Medical Entity Recognition in Chinese Clinical Text.

Negation Detection for Clinical Text Mining in Russian

A comprehensive study of named entity recognition in Chinese clinical text

Natural language processing to extract medical problems from electronic clinical documents: Performance evaluation

A Large Language Model to Detect Negated Expressions in Radiology Reports

Extracting Important Information From Chinese Operation Notes With Natural Language Processing Methods

Improving Chinese Electronic Medical Record Retrieval by Field Weight Assignment, Negation Detection, and Re-ranking

Deep Learning Approach for Negation and Speculation Detection for Automated Important Finding Flagging and Extraction in Radiology Report: Internal Validation and Technique Comparison Study

Using Natural Language Processing to Extract Clinically Useful Information from Chinese Electronic Medical Records

A Fine-Grained Chinese Word Segmentation and Part-of-speech Tagging Corpus for Clinical Text

A query interface for clinical research with Chinese electronic health record using Natural Language Processing