Utilizing a digital swarm intelligence platform to improve consensus among radiologists and exploring its applications
Rutwik Shah,Bruno Astuto,Tyler Gleason,Will Fletcher,Justin Banaga,Kevin Sweetwood,Allen Ye,Rina Patel,Kevin McGill,Thomas Link,Jason Crane,Valentina Pedoia,Sharmila Majumdar
DOI: https://doi.org/10.48550/arXiv.2107.07341
2021-09-07
Abstract:Radiologists today play a key role in making diagnostic decisions and labeling images for training A.I. algorithms. Low inter-reader reliability (IRR) can be seen between experts when interpreting challenging cases. While teams-based decisions are known to outperform individual decisions, inter-personal biases often creep up in group interactions which limit non-dominant participants from expressing true opinions. To overcome the dual problems of low consensus and inter-personal bias, we explored a solution modeled on biological swarms of bees. Two separate cohorts; three radiologists and five radiology residents collaborated on a digital swarm platform in real time and in a blinded fashion, grading meniscal lesions on knee MR exams. These consensus votes were benchmarked against clinical (arthroscopy) and radiological (senior-most radiologist) observations. The IRR of the consensus votes was compared to the IRR of the majority and most confident votes of the two <a class="link-external link-http" href="http://cohorts.The" rel="external noopener nofollow">this http URL</a> radiologist cohort saw an improvement of 23% in IRR of swarm votes over majority vote. Similar improvement of 23% in IRR in 3-resident swarm votes over majority vote, was observed. The 5-resident swarm had an even higher improvement of 32% in IRR over majority vote. Swarm consensus votes also improved specificity by up to 50%. The swarm consensus votes outperformed individual and majority vote decisions in both the radiologists and resident cohorts. The 5-resident swarm had higher IRR than 3-resident swarm indicating positive effect of increased swarm size. The attending and resident swarms also outperformed predictions from a state-of-the-art A.I. algorithm. Utilizing a digital swarm platform improved agreement and allows participants to express judgement free intent, resulting in superior clinical performance and robust A.I. training labels.
Human-Computer Interaction,Artificial Intelligence,Distributed, Parallel, and Cluster Computing,Machine Learning,Neural and Evolutionary Computing,Social and Information Networks