Abstract:Adapting foundation models for medical image analysis requires finetuning them on a considerable amount of data because of extreme distribution shifts between natural (source) data used for pretraining and medical (target) data. However, collecting task-specific medical data for such finetuning at a central location raises many privacy concerns. Although Federated learning (FL) provides an effective means for training on private decentralized data, communication costs in federating large foundation models can quickly become a significant bottleneck, impacting the solution's scalability. In this work, we address this problem of efficient communication while ensuring effective learning in FL by combining the strengths of Parameter-Efficient Fine-tuning (PEFT) with FL. Specifically, we study plug-and-play Low-Rank Adapters (LoRA) in a federated manner to adapt the Segment Anything Model (SAM) for 3D medical image segmentation. Unlike prior works that utilize LoRA and finetune the entire decoder, we critically analyze the contribution of each granular component of SAM on finetuning performance. Thus, we identify specific layers to be federated that are very efficient in terms of communication cost while producing on-par accuracy. Our experiments show that retaining the parameters of the SAM model (including most of the decoder) in their original state during adaptation is beneficial because fine-tuning on small datasets tends to distort the inherent capabilities of the underlying foundation model. On Fed-KiTS, our approach decreases communication cost (~48x) compared to full fine-tuning while increasing performance (~6% Dice score) in 3D segmentation tasks. Our approach performs similar to SAMed while achieving ~2.8x reduction in communication and parameters to be finetuned. We further validate our approach with experiments on Fed-IXI and Prostate MRI datasets.

Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model

Integrating Spatial Prior Adapter for Enhancing SAM Performance in Medical Image Segmentation

Task-Aware Low-Rank Adaptation of Segment Anything Model

A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation

Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few Exemplars

SU-SAM: A Simple Unified Framework for Adapting Segment Anything Model in Underperformed Scenes

SAM-Adapter: Adapting Segment Anything in Underperformed Scenes

AM-SAM: Automated Prompting and Mask Calibration for Segment Anything Model

SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors

SA-FedLora: Adaptive Parameter Allocation for Efficient Federated Learning with LoRA Tuning

SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More

Concatenate, Fine-tuning, Re-training: A SAM-enabled Framework for Semi-supervised 3D Medical Image Segmentation

MeSAM: Multiscale Enhanced Segment Anything Model for Optical Remote Sensing Images

Customized Segment Anything Model for Medical Image Segmentation

Tuning a SAM-Based Model with Multi-Cognitive Visual Adapter to Remote Sensing Instance Segmentation

Ladder Fine-tuning approach for SAM integrating complementary network

BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAM

GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation