EFCM: Efficient Fine-tuning on Compressed Models for deployment of large models in medical image analysis

Shaojie Li,Zhaoshuo Diao

2024-09-18

Abstract:The recent development of deep learning large models in medicine shows remarkable performance in medical image analysis and diagnosis, but their large number of parameters causes memory and inference latency challenges. Knowledge distillation offers a solution, but the slide-level gradients cannot be backpropagated for student model updates due to high-resolution pathological images and slide-level labels. This study presents an Efficient Fine-tuning on Compressed Models (EFCM) framework with two stages: unsupervised feature distillation and fine-tuning. In the distillation stage, Feature Projection Distillation (FPD) is proposed with a TransScan module for adaptive receptive field adjustment to enhance the knowledge absorption capability of the student model. In the slide-level fine-tuning stage, three strategies (Reuse CLAM, Retrain CLAM, and End2end Train CLAM (ETC)) are compared. Experiments are conducted on 11 downstream datasets related to three large medical models: RETFound for retina, MRM for chest X-ray, and BROW for histopathology. The experimental results demonstrate that the EFCM framework significantly improves accuracy and efficiency in handling slide-level pathological image problems, effectively addressing the challenges of deploying large medical models. Specifically, it achieves a 4.33% increase in ACC and a 5.2% increase in AUC compared to the large model BROW on the TCGA-NSCLC and TCGA-BRCA datasets. The analysis of model inference efficiency highlights the high efficiency of the distillation fine-tuning method.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the deployment challenges of large - scale deep - learning models in medical image analysis. Specifically, although large - scale models perform excellently in medical image analysis and diagnosis, their large number of parameters leads to high memory consumption and long inference latency. In addition, the existing knowledge distillation methods have limitations when dealing with high - resolution pathological images and slide - level labels, because the slide - level gradients cannot be back - propagated in parallel to the feature encoder to update the student model.

To solve these problems, the paper proposes a framework named Efficient Fine - tuning on Compressed Models (EFCM), which achieves efficient compression and fine - tuning through two stages:

1. **Unsupervised Feature Distillation**: In this stage, the paper proposes a method named Feature Projection Distillation (FPD), combined with a TransScan module for adaptively adjusting the receptive field size, thereby enhancing the knowledge absorption ability of the student model.
2. **Slide - level Fine - tuning**: In this stage, the paper compares three strategies (Reuse CLAM, Retrain CLAM, and End2end Train CLAM (ETC)) to evaluate the performance of the student model after distillation.

Through these two stages, the EFCM framework aims to significantly improve the accuracy and efficiency of the model, especially when dealing with slide - level pathological image classification problems. The experimental results show that, compared with large - scale models, the EFCM framework significantly improves the accuracy rate and AUC value on multiple downstream data sets, and also performs excellently in terms of inference efficiency.