Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components

Hermione Warr,Yasin Ibrahim,Daniel R. McGowan,Konstantinos Kamnitsas
2024-07-31
Abstract:Automation of medical image interpretation could alleviate bottlenecks in diagnostic workflows, and has become of particular interest in recent years due to advancements in natural language processing. Great strides have been made towards automated radiology report generation via AI, yet ensuring clinical accuracy in generated reports is a significant challenge, hindering deployment of such methods in clinical practice. In this work we propose a quality control framework for assessing the reliability of AI-generated radiology reports with respect to semantics of diagnostic importance using modular auxiliary auditing components (AC). Evaluating our pipeline on the MIMIC-CXR dataset, our findings show that incorporating ACs in the form of disease-classifiers can enable auditing that identifies more reliable reports, resulting in higher F1 scores compared to unfiltered generated reports. Additionally, leveraging the confidence of the AC labels further improves the audit's effectiveness.
Artificial Intelligence,Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to ensure clinical accuracy in automatically - generated radiology reports. Specifically, the paper proposes a quality - control framework to evaluate the reliability of AI - generated radiology reports in terms of diagnostic importance through Auxiliary Audit Components (ACs). This is mainly because, although significant progress has been made in automatic radiology report generation, ensuring the clinical accuracy of the generated reports remains a major challenge, which hinders the deployment of these methods in clinical practice. The paper mentions that existing language models face multiple challenges when applied to medical scenarios, and one of the key issues is the need to ensure factual accuracy. To address this challenge, the authors introduce a quality - control framework based on modular Auxiliary Audit Components to identify potential errors in AI - generated radiology reports. These Auxiliary Audit Components are mainly used to extract diagnosis - related semantic information from images and compare it with the content of the generated reports to check the reliability and accuracy of the reports. Through this method, the paper aims to improve the quality of AI - generated radiology reports, make them more in line with clinical needs, and thus promote the application of these technologies in the actual medical environment.