Abstract:Volumetric medical image segmentation is pivotal in enhancing disease diagnosis, treatment planning, and advancing medical research. While existing volumetric foundation models for medical image segmentation, such as SAM-Med3D and SegVol, have shown remarkable performance on general organs and tumors, their ability to segment certain categories in clinical downstream tasks remains limited. Supervised Finetuning (SFT) serves as an effective way to adapt such foundation models for task-specific downstream tasks but at the cost of degrading the general knowledge previously stored in the original foundation <a class="link-external link-http" href="http://model.To" rel="external noopener nofollow">this http URL</a> address this, we propose SAM-Med3D-MoE, a novel framework that seamlessly integrates task-specific finetuned models with the foundational model, creating a unified model at minimal additional training expense for an extra gating network. This gating network, in conjunction with a selection strategy, allows the unified model to achieve comparable performance of the original models in their respective tasks both general and specialized without updating any parameters of them.Our comprehensive experiments demonstrate the efficacy of SAM-Med3D-MoE, with an average Dice performance increase from 53 to 56.4 on 15 specific classes. It especially gets remarkable gains of 29.6, 8.5, 11.2 on the spinal cord, esophagus, and right hip, respectively. Additionally, it achieves 48.9 Dice on the challenging SPPIN2023 Challenge, significantly surpassing the general expert's performance of 32.3. We anticipate that SAM-Med3D-MoE can serve as a new framework for adapting the foundation model to specific areas in medical image analysis. Codes and datasets will be publicly available.

Instance Segmentation of Mouse Brain Scanning Electron Microscopy Images Based on Fine-Tuning Nature Image Model

ShapeMamba-EM: Fine-Tuning Foundation Model with Local Shape Descriptors and Mamba Blocks for 3D EM Image Segmentation

Segmentation Of Mitochondria Based On Sem Images

Hierarchical Level Features Based Trainable Segmentation for Electron Microscopy Images

Semi-Automatic Neuron Tracing in Electron Microscope Images: Preliminary Results

A Hybrid Cnn-Rf Method for Electron Microscopy Images Segmentation

Segment Anything for Dendrites from Electron Microscopy

SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentation

How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model

DEEP CONTEXTUAL RESIDUAL NETWORK FOR ELECTRON MICROSCOPY IMAGE SEGMENTATION IN CONNECTOMICS

Accuracy of Segment-Anything Model (SAM) in medical image segmentation tasks

Automatically Segmenting and Reconstructing Neurons in SEM Images

The 3D Instance Segmentation Network for Synapse Reconstruction from Serial Electron Microscopy Images

Empirical Evaluation of the Segment Anything Model (SAM) for Brain Tumor Segmentation

Improving the Neural Segmentation of Blurry Serial SEM Images by Blind Deblurring

Enhancing Neural Segmentation on Low-Quality SEM Images Using Self-Attention Based Fusion Module with Auxiliary Deblurring Network

A prior feature SVM-MRF based method for mouse brain segmentation.

A modular hierarchical approach to 3D electron microscopy image segmentation.

Adapting Segment Anything Model (SAM) through Prompt-based Learning for Enhanced Protein Identification in Cryo-EM Micrographs

nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance

Learning the Heterogeneous Representation of Brain's Structure from Serial SEM Images Using a Masked Autoencoder