Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data

Chaoyi Wu,Xiaoman Zhang,Ya Zhang,Yanfeng Wang,Weidi Xie

2023-11-16

Abstract:In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM. We consider the construction of foundational models from three perspectives, namely, dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM outperforms existing multi-modal foundation models, that are publicaly accessible, including Openflamingo, MedFlamingo, MedVInT and GPT-4V. Additionally, we also adapt RadFM for different public benchmarks, surpassing existing SOTAs on diverse datasets. All codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

Computer Vision and Pattern Recognition,Computation and Language

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on developing a general Radiology Foundation Model (RadFM) to address three key challenges in the current medical field:

1. **Lack of multi - modal datasets**: Medical data is essentially multi - modal, including text (electronic health records, medical reports), 1D signals (electrocardiograms), 2D images (ultrasounds, X - rays), 3D images (CT or MRI scans), etc. To support the training of a general medical foundation model, a large - scale, diverse multi - modal dataset is required.

2. **Lack of general architecture design**: In the existing AI4Medicine literature, different clinical tasks are usually solved through the "divide - and - conquer" paradigm, that is, designing different architectures for each specific problem. However, developing a general medical foundation model requires a unified architecture that can handle multiple clinical tasks and fuse information from different modalities.

3. **Lack of effective evaluation benchmarks**: Currently, the benchmark tests used to evaluate the clinical knowledge of models mainly rely on task - specific datasets, and these datasets have a limited number of test cases. Therefore, there is an urgent need to establish a high - quality evaluation benchmark to comprehensively measure the progress of medical foundation models in a wide range of clinical tasks.

To solve these problems, the authors have taken the following measures:

- **Construct a large - scale multi - modal dataset**: They created a large - scale medical multi - modal dataset named MedMD, which contains 16 million 2D and 3D medical scan images and high - quality text descriptions, covering more than 5,000 diseases. In addition, a subset RadMD specifically for the radiology field was also constructed, which contains 3 million screened multi - modal samples.

- **Propose a new model architecture**: RadFM is a visual - conditional autoregressive text generation model that can seamlessly integrate natural language with 2D or 3D medical scans and solve a wide range of medical tasks through natural language output. This model is first pre - trained on MedMD and then fine - tuned on RadMD.

- **Establish a comprehensive evaluation benchmark**: The authors introduced a new evaluation benchmark, RadBench, which covers five tasks: modality identification, disease diagnosis, visual question answering, report generation, and reasoning diagnosis, aiming to comprehensively evaluate the performance of the foundation model in actual clinical problems.

Through the above work, the authors have demonstrated the potential for developing a general radiology foundation model and provided important resources and tools for future research.