Collaboration between clinicians and vision–language models in radiology report generation
Ryutaro Tanno,David G. T. Barrett,Andrew Sellergren,Sumedh Ghaisas,Sumanth Dathathri,Abigail See,Johannes Welbl,Charles Lau,Tao Tu,Shekoofeh Azizi,Karan Singhal,Mike Schaekermann,Rhys May,Roy Lee,SiWai Man,Sara Mahdavi,Zahra Ahmed,Yossi Matias,Joelle Barral,S. M. Ali Eslami,Danielle Belgrave,Yun Liu,Sreenivasa Raju Kalidindi,Shravya Shetty,Vivek Natarajan,Pushmeet Kohli,Po-Sen Huang,Alan Karthikesalingam,Ira Ktena
DOI: https://doi.org/10.1038/s41591-024-03302-1
IF: 82.9
2024-11-08
Nature Medicine
Abstract:Automated radiology report generation has the potential to improve patient care and reduce the workload of radiologists. However, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of artificial intelligence (AI)-generated reports. We build a state-of-the-art report generation system for chest radiographs, called Flamingo-CXR, and perform an expert evaluation of AI-generated reports by engaging a panel of board-certified radiologists. We observe a wide distribution of preferences across the panel and across clinical settings, with 56.1% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for in/outpatient X-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. Errors were observed in human-written reports and Flamingo-CXR reports, with 24.8% of in/outpatient cases containing clinically significant errors in both report types, 22.8% in Flamingo-CXR reports only and 14.0% in human reports only. For reports that contain errors we develop an assistive setting, a demonstration of clinician–AI collaboration for radiology report composition, indicating new possibilities for potential clinical utility.
biochemistry & molecular biology,cell biology,medicine, research & experimental