Abstract:Despite the rapid progress in self-supervised learning (SSL), end-to-end fine-tuning still remains the dominant fine-tuning strategy for medical imaging analysis. However, it remains unclear whether this approach is truly optimal for effectively utilizing the pre-trained knowledge, especially considering the diverse categories of SSL that capture different types of features. In this paper, we present the first comprehensive study that discovers effective fine-tuning strategies for self-supervised learning in medical imaging. After developing strong contrastive and restorative SSL baselines that outperform SOTA methods across four diverse downstream tasks, we conduct an extensive fine-tuning analysis across multiple pre-training and fine-tuning datasets, as well as various fine-tuning dataset sizes. Contrary to the conventional wisdom of fine-tuning only the last few layers of a pre-trained network, we show that fine-tuning intermediate layers is more effective, with fine-tuning the second quarter (25-50%) of the network being optimal for contrastive SSL whereas fine-tuning the third quarter (50-75%) of the network being optimal for restorative SSL. Compared to the de-facto standard of end-to-end fine-tuning, our best fine-tuning strategy, which fine-tunes a shallower network consisting of the first three quarters (0-75%) of the pre-trained network, yields improvements of as much as 5.48%. Additionally, using these insights, we propose a simple yet effective method to leverage the complementary strengths of multiple SSL models, resulting in enhancements of up to 3.57% compared to using the best model alone. Hence, our fine-tuning strategies not only enhance the performance of individual SSL models, but also enable effective utilization of the complementary strengths offered by multiple SSL models, leading to significant improvements in self-supervised medical imaging analysis.

Contrastive-Adversarial and Diffusion: Exploring pre-training and fine-tuning strategies for sulcal identification

Align Yourself: Self-supervised Pre-training for Fine-grained Recognition via Saliency Alignment.

Contrastive Learning to Fine-Tune Feature Extraction Models for the Visual Cortex

A Study on Training Fine-Tuning of Convolutional Neural Networks.

CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning

An Analysis of Unsupervised Pre-training in Light of Recent Advances

Continual Learning with Pretrained Backbones by Tuning in the Input Space

Multi-level Asymmetric Contrastive Learning for Volumetric Medical Image Segmentation Pre-training

What makes for good views for contrastive learning

LEARNER: Learning Granular Labels from Coarse Labels using Contrastive Learning

Resource and data efficient self supervised learning

Comparison of fine-tuning strategies for transfer learning in medical image classification

Influence of contrast and texture based image modifications on the performance and attention shift of U-Net models for brain tissue segmentation

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Learning to segment with limited annotations: Self-supervised pretraining with regression and contrastive loss in MRI

PARTICLE: Part Discovery and Contrastive Learning for Fine-grained Recognition

What Makes for Good Views for Contrastive Learning?

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Revisiting Fine-Tuning Strategies for Self-supervised Medical Imaging Analysis