Abstract:The widespread adoption of large-scale pre-training techniques has significantly advanced the development of medical foundation models, enabling them to serve as versatile tools across a broad range of medical tasks. However, despite their strong generalization capabilities, medical foundation models pre-trained on large-scale datasets tend to suffer from domain gaps between heterogeneous data, leading to suboptimal performance on specific tasks compared to specialist models, as evidenced by previous studies. In this paper, we explore a new perspective called "Knowledge Decomposition" to improve the performance on specific medical tasks, which deconstructs the foundation model into multiple lightweight expert models, each dedicated to a particular anatomical region, with the aim of enhancing specialization and simultaneously reducing resource consumption. To accomplish the above objective, we propose a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates gradients from different tasks by incorporating low-rank expert modules and efficient knowledge separation convolution. The low-rank expert modules resolve gradient conflicts between heterogeneous data from different anatomical regions, providing strong specialization at lower costs. The efficient knowledge separation convolution significantly improves algorithm efficiency by achieving knowledge separation within a single forward propagation. Extensive experimental results on segmentation and classification tasks demonstrate that our decomposed models not only achieve state-of-the-art performance but also exhibit superior transferability on downstream tasks, even surpassing the original foundation models in task-specific evaluations. The code is available at here.

What problem does this paper attempt to address?

This paper attempts to solve the following two main problems: 1. **Performance degradation due to domain gap**: Although large - scale pre - training techniques have significantly promoted the development of medical basic models and made them perform well in a wide range of medical tasks, these models often perform less well than specialized models when dealing with specific tasks. This is because the significant anatomical differences between different human body regions (such as the abdomen and the brain) lead to a large gap in the data domain. This gap makes pre - training models face challenges when dealing with heterogeneous data, thus sacrificing performance in specific regions. 2. **High consumption of computing resources**: Due to their large number of parameters and high computing requirements, medical basic models become impractical to be deployed in resource - constrained medical environments. For example, the requirement for high - performance hardware makes it difficult for many hospitals, especially those in under - developed areas, to afford. To address these problems, the authors propose a new method - **Low - Rank Knowledge Decomposition (LoRKD)**, which aims to improve the performance of specific medical tasks and reduce resource consumption by decomposing the basic model into multiple lightweight expert models. Specifically, LoRKD realizes the explicit gradient separation of tasks in different anatomical regions by introducing low - rank expert modules and efficient knowledge - separating convolutions, thus solving the gradient conflict between heterogeneous data and improving the specialization accuracy and efficiency of the model. ### Main contributions - **Knowledge decomposition**: In view of the data heterogeneity in the medical field, the concept of knowledge decomposition is introduced, and the basic model is decomposed into multiple lightweight expert models to reduce costs and enhance specialization accuracy. - **Novel framework**: A new framework LoRKD, which includes low - rank expert modules and efficient knowledge - separating convolutions, is proposed. Task - specific knowledge is injected into the corresponding expert modules through efficient explicit gradient separation. - **Superior performance**: Extensive experimental results show that LoRKD can not only decompose the basic model into lighter and stronger expert models, but also achieve better specialization accuracy in segmentation and classification tasks and the transfer ability of downstream tasks. Through these improvements, LoRKD not only improves the performance of the model on specific tasks, but also significantly reduces the demand for computing resources, making these models more suitable for practical deployment.

LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models

Low-Rank Knowledge Decomposition for Medical Foundation Models

Adaptive Decomposition and Shared Weight Volumetric Transformer Blocks for Efficient Patch-Free 3D Medical Image Segmentation.

KA^2ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation

Reprogramming Distillation for Medical Foundation Models

FKD-Med: Privacy-Aware, Communication-Optimized Medical Image Segmentation via Federated Learning and Model Lightweighting Through Knowledge Distillation

MSKD: Structured knowledge distillation for efficient medical image segmentation

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

KA$^2$ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation

SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

Decompose-and-Integrate Learning for Multi-class Segmentation in Medical Images

A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis

Foundation Ark: Accruing and Reusing Knowledge for Superior and Robust Performance

Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models

KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation

Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision

Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition

MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

GrOD : Deep Learning with Gradients Orthogonal Decomposition for Knowledge Transfer, Distillation, and Adversarial Training

RKSeg+: make full use of Runge–Kutta methods in medical image segmentation

Adaptive Affinity-Based Generalization For MRI Imaging Segmentation Across Resource-Limited Settings