Abstract:The advancement of artificial intelligence (AI) for organ segmentation and tumor detection is propelled by the growing availability of computed tomography (CT) datasets with detailed, per-voxel annotations. However, these AI models often struggle with flexibility for partially annotated datasets and extensibility for new classes due to limitations in the one-hot encoding, architectural design, and learning scheme. To overcome these limitations, we propose a universal, extensible framework enabling a single model, termed Universal Model, to deal with multiple public datasets and adapt to new classes (e.g., organs/tumors). Firstly, we introduce a novel language-driven parameter generator that leverages language embeddings from large language models, enriching semantic encoding compared with one-hot encoding. Secondly, the conventional output layers are replaced with lightweight, class-specific heads, allowing Universal Model to simultaneously segment 25 organs and six types of tumors and ease the addition of new classes. We train our Universal Model on 3,410 CT volumes assembled from 14 publicly available datasets and then test it on 6,173 CT volumes from four external datasets. Universal Model achieves first place on six CT tasks in the Medical Segmentation Decathlon (MSD) public leaderboard and leading performance on the Beyond The Cranial Vault (BTCV) dataset. In summary, Universal Model exhibits remarkable computational efficiency (6x faster than other dataset-specific models), demonstrates strong generalization across different hospitals, transfers well to numerous downstream tasks, and more importantly, facilitates the extensibility to new classes while alleviating the catastrophic forgetting of previously learned classes. Codes, models, and datasets are available at <a class="link-external link-https" href="https://github.com/ljwztc/CLIP-Driven-Universal-Model" rel="external noopener nofollow">this https URL</a>

Large Foundation Model for Cancer Segmentation

Foundation model for cancer imaging biomarkers

Foundation AI Model for Medical Image Segmentation

On the Challenges and Perspectives of Foundation Models for Medical Image Analysis

Empirical Analysis of a Segmentation Foundation Model in Prostate Imaging

Large-scale foundation model on single-cell transcriptomics

A Large Language Model Pipeline for Breast Cancer Oncology

CancerLLM: A Large Language Model in Cancer Domain

Utilizing large language models in breast cancer management: systematic review

Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography

Applications of large language models in cancer care: current evidence and future perspectives

Large-scale Foundation Models and Generative AI for BigData Neuroscience

Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images

In-context learning enables multimodal large language models to classify cancer pathology images

Large scale models in radiology: revolutionizing the future of medical imaging

Foundation Models for Slide-level Cancer Subtyping in Digital Pathology

Foundation Models in Radiology: What, How, When, Why and Why Not

Diagnosis Assistant for Liver Cancer Utilizing a Large Language Model with Three Types of Knowledge

A foundation model for clinical-grade computational pathology and rare cancers detection

CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection