Abstract:Background: There is progress to be made in building artificially intelligent systems to detect abnormalities that are not only accurate but can handle the true breadth of findings that radiologists encounter in body (chest, abdomen, and pelvis) computed tomography (CT). Currently, the major bottleneck for developing multi-disease classifiers is a lack of manually annotated data. The purpose of this work was to develop high throughput multi-label annotators for body CT reports that can be applied across a variety of abnormalities, organs, and disease states thereby mitigating the need for human annotation. Methods: We used a dictionary approach to develop rule-based algorithms (RBA) for extraction of disease labels from radiology text reports. We targeted three organ systems (lungs/pleura, liver/gallbladder, kidneys/ureters) with four diseases per system based on their prevalence in our dataset. To expand the algorithms beyond pre-defined keywords, attention-guided recurrent neural networks (RNN) were trained using the RBA-extracted labels to classify reports as being positive for one or more diseases or normal for each organ system. Alternative effects on disease classification performance were evaluated using random initialization or pre-trained embedding as well as different sizes of training datasets. The RBA was tested on a subset of 2158 manually labeled reports and performance was reported as accuracy and F-score. The RNN was tested against a test set of 48,758 reports labeled by RBA and performance was reported as area under the receiver operating characteristic curve (AUC), with 95% CIs calculated using the DeLong method. Results: Manual validation of the RBA confirmed 91-99% accuracy across the 15 different labels. Our models extracted disease labels from 261,229 radiology reports of 112,501 unique subjects. Pre-trained models outperformed random initialization across all diseases. As the training dataset size was reduced, performance was robust except for a few diseases with a relatively small number of cases. Pre-trained classification AUCs reached > 0.95 for all four disease outcomes and normality across all three organ systems. Conclusions: Our label-extracting pipeline was able to encompass a variety of cases and diseases in body CT reports by generalizing beyond strict rules with exceptional accuracy. The method described can be easily adapted to enable automated labeling of hospital-scale medical data sets for training image-based disease classifiers.

CheX-GPT: Harnessing Large Language Models for Enhanced Chest X-ray Report Labeling

German CheXpert Chest X-ray Radiology Report Labeler

Language model-based labeling of German thoracic radiology reports

Effect of Radiology Report Labeler Quality on Deep Learning Models for Chest X-Ray Interpretation

Multi-Label Annotation of Chest Abdomen Pelvis Computed Tomography Text Reports Using Deep Learning

CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats

A Disease Labeler for Chinese Chest X-Ray Report Generation

Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification

Comparing Commercial and Open-Source Large Language Models for Labeling Chest Radiograph Reports

Highly accurate classification of chest radiographic reports using a deep learning natural language model pre-trained on 3.8 million text reports

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Multi-label annotation of text reports from computed tomography of the chest, abdomen, and pelvis using deep learning

Automated labelling of radiology reports using natural language processing: Comparison of traditional and newer methods

XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models

Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reports

Automated Spinal MRI Labelling from Reports Using a Large Language Model

Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential

Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential

Extracting and Learning Fine-Grained Labels from Chest Radiographs

Learning to diagnose common thorax diseases on chest radiographs from radiology reports in Vietnamese