Evaluating progress in automatic chest X-ray radiology report generation

Feiyang Yu,Mark Endo,Rayan Krishnan,Ian Pan,Andy Tsai,Eduardo Pontes Reis,Eduardo Kaiser Ururahy Nunes Fonseca,Henrique Min Ho Lee,Zahra Shakeri Hossein Abad,Andrew Y. Ng,Curtis P. Langlotz,Vasantha Kumar Venugopal,Pranav Rajpurkar
DOI: https://doi.org/10.1016/j.patter.2023.100802
2023-08-05
Patterns
Abstract:Summary Artificial intelligence (AI) models for automatic generation of narrative radiology reports from images have the potential to enhance efficiency and reduce the workload of radiologists. However, evaluating the correctness of these reports requires metrics that can capture clinically pertinent differences. In this study, we investigate the alignment between automated metrics and radiologists' scoring of errors in report generation. We address the limitations of existing metrics by proposing new metrics, RadGraph F1 and RadCliQ, which demonstrate stronger correlation with radiologists' evaluations. In addition, we analyze the failure modes of the metrics to understand their limitations and provide guidance for metric selection and interpretation. This study establishes RadGraph F1 and RadCliQ as meaningful metrics for guiding future research in radiology report generation.
What problem does this paper attempt to address?