Abstract:Purpose: To perform an in-depth evaluation of current state of the art techniques in training neural networks to identify appropriate approaches in small datasets. Method: In total, 112,120 frontal-view X-ray images from the NIH ChestXray14 dataset were used in our analysis. Two tasks were studied: unbalanced multi-label classification of 14 diseases, and binary classification of pneumonia vs non-pneumonia. All datasets were randomly split into training, validation, and testing (70%, 10%, and 20%). Two popular convolution neural networks (CNNs), DensNet121 and ResNet50, were trained using PyTorch. We performed several experiments to test: (a) whether transfer learning using pretrained networks on ImageNet are of value to medical imaging/physics tasks (e.g., predicting toxicity from radiographic images after training on images from the internet), (b) whether using pretrained networks trained on problems that are similar to the target task helps transfer learning (e.g., using X-ray pretrained networks for X-ray target tasks), (c) whether freeze deep layers or change all weights provides an optimal transfer learning strategy, (d) the best strategy for the learning rate policy, and (e) what quantity of data is needed in order to appropriately deploy these various strategies (N = 50 to N = 77 880). Results: In the multi-label problem, DensNet121 needed at least 1600 patients to be comparable to, and 10 000 to outperform, radiomics-based logistic regression. In classifying pneumonia vs non-pneumonia, both CNN and radiomics-based methods performed poorly when N < 2000. For small datasets ( < 2000), however, a significant boost in performance (>15% increase on AUC) comes from a good selection of the transfer learning dataset, dropout, cycling learning rate, and freezing and unfreezing of deep layers as training progresses. In contrast, if sufficient data are available (>35 000), little or no tweaking is needed to obtain impressive performance. While transfer learning using X-ray images from other anatomical sites improves performance, we also observed a similar boost by using pretrained networks from ImageNet. Having source images from the same anatomical site, however, outperforms every other methodology, by up to 15%. In this case, DL models can be trained with as little as N = 50. Conclusions: While training DL models in small datasets (N < 2000) is challenging, no tweaking is necessary for bigger datasets (N > 35 000). Using transfer learning with images from the same anatomical site can yield remarkable performance in new tasks with as few as N = 50. Surprisingly, we did not find any advantage for using images from other anatomical sites over networks that have been trained using ImageNet. This indicates that features learned may not be as general as currently believed, and performance decays rapidly even by just changing the anatomical site of the images.

Comparison of fine-tuning strategies for transfer learning in medical image classification

Comparison of fine-tuning strategies for transfer learning in medical image classification

Fine-tuning pre-trained neural networks for medical image classification in small clinical datasets

How Deeply to Fine-Tune a Convolutional Neural Network: A Case Study Using a Histopathology Dataset

Revisiting Fine-Tuning Strategies for Self-supervised Medical Imaging Analysis

Effect of layer-wise fine-tuning in magnification-dependent classification of breast cancer histopathological image

Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging

Exploring a Universal Training Method for Medical Image Classification.

Transfer learning for medical image classification: a literature review

On the Effective Transfer Learning Strategy for Medical Image Analysis in Deep Learning

Enhancing Alzheimer's Disease Classification with Transfer Learning: Finetuning a Pre-trained Algorithm

Rethinking Transfer Learning for Medical Image Classification

Why does my medical AI look at pictures of birds? Exploring the efficacy of transfer learning across domain boundaries

Transfer learning from a sparsely annotated dataset of 3D medical images

Targeted transfer learning to improve performance in small medical physics datasets

Enhanced Transfer Learning Through Medical Imaging and Patient Demographic Data Fusion

Beyond fine-tuning: Classifying high resolution mammograms using function-preserving transformations

A scoping review of transfer learning research on medical image analysis using ImageNet

Transfer Learning With Adaptive Fine-Tuning

Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity

Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging