Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation

Yunhe Gao,Zhuowei Li,Di Liu,Mu Zhou,Shaoting Zhang,Dimitris N. Metaxas

2024-04-07

Abstract:A major focus of clinical imaging workflow is disease diagnosis and management, leading to medical imaging datasets strongly tied to specific clinical objectives. This scenario has led to the prevailing practice of developing task-specific segmentation models, without gaining insights from widespread imaging cohorts. Inspired by the training program of medical radiology residents, we propose a shift towards universal medical image segmentation, a paradigm aiming to build medical image understanding foundation models by leveraging the diversity and commonality across clinical targets, body regions, and imaging modalities. Towards this goal, we develop Hermes, a novel context-prior learning approach to address the challenges of data heterogeneity and annotation differences in medical image segmentation. In a large collection of eleven diverse datasets (2,438 3D images) across five modalities (CT, PET, T1, T2 and cine MRI) and multiple body regions, we demonstrate the merit of the universal paradigm over the traditional paradigm on addressing multiple tasks within a single model. By exploiting the synergy across tasks, Hermes achieves state-of-the-art performance on all testing datasets and shows superior model scalability. Results on two additional datasets reveals Hermes' strong performance for transfer learning, incremental learning, and generalization to downstream tasks. Hermes's learned priors demonstrate an appealing trait to reflect the intricate relations among tasks and modalities, which aligns with the established anatomical and imaging principles in radiology. The code is available: <a class="link-external link-https" href="https://github.com/yhygao/universal-medical-image-segmentation" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper proposes a new medical image segmentation method called Hermes to address the limitations of existing task-specific models. The current practice is to develop separate segmentation models for each medical target or imaging modality, which limits the model's generalization ability and the insights from a wide imaging queue. Hermes achieves universal medical image segmentation by learning contextual prior knowledge, handling data heterogeneity and annotation discrepancies. It aims to build a foundational model capable of understanding different clinical targets, body regions, and imaging modalities. Experimental results demonstrate that Hermes performs well on multiple tasks, exhibiting better model scalability and strong performance in transfer learning, incremental learning, and downstream task generalization.