Abstract:Increasing model, data and compute budget scale in the pre-training has been shown to strongly improve model generalization and transfer learning in vast line of work done in language modeling and natural image recognition. However, most studies on the positive effect of larger scale were done in scope of in-domain setting, with source and target data being in close proximity. To study effect of larger scale for both in-domain and out-of-domain setting when performing full and few-shot transfer, we combine here for the first time large, openly available medical X-Ray chest imaging datasets to reach a scale for medical imaging domain comparable to ImageNet-1k, routinely used for pre-training in natural image domain. We then conduct supervised pre-training, while varying network size and source data scale and domain, being either large natural (ImageNet-1k/21k) or large medical chest X-Ray datasets, and transfer pre-trained models to different natural or medical targets. We observe strong improvement due to larger pre-training scale for intra-domain natural-natural and medical-medical transfer. For inter-domain natural-medical transfer, we find improvements due to larger pre-training scale on larger X-Ray targets in full shot regime, while for smaller targets and for few-shot regime the improvement is not visible. Remarkably, large networks pre-trained on very large natural ImageNet-21k are as good or better than networks pre-trained on largest available medical X-Ray data when performing transfer to large X-Ray targets. We conclude that substantially increasing model and generic, medical domain-agnostic natural image source data scale in the pre-training can enable high quality out-of-domain transfer to medical domain specific targets, removing dependency on large medical domain-specific source data often not available in the practice.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: to study the influence of the scale of pre - training models (including the number of model parameters, the size of data sets and computing resources) on the effect of full - sample and few - sample transfer learning in intra - domain and inter - domain transfer learning of natural images and medical X - ray chest images. Specifically, the authors focus on the following aspects: 1. **Intra - domain transfer learning**: - Between natural images to natural images (such as ImageNet to CIFAR - 10) and medical X - ray images to medical X - ray images (such as CheXpert to MIMIC - CXR), whether the increase in pre - training scale can improve the effect of transfer learning. 2. **Inter - domain transfer learning**: - Between natural images to medical X - ray images (such as ImageNet to CheXpert), whether the increase in pre - training scale can improve the effect of transfer learning. 3. **Full - sample and few - sample transfer learning**: - To study the different influences of pre - training scale on full - sample (full - shot) and few - sample (few - shot) transfer learning. In particular, in the case of only a small amount of labeled data, whether the increase in pre - training scale can significantly improve the model performance. ### Main findings 1. **Improvement of intra - domain transfer learning**: - Whether it is from natural images to natural images or from medical X - ray images to medical X - ray images, with the increase in the scale of pre - training models and data, the transfer learning performance has been significantly improved. Especially in the transfer from natural images to natural images, this improvement is very obvious in both full - sample and few - sample transfer learning. 2. **Complexity of inter - domain transfer learning**: - For the cross - domain transfer from natural images to medical X - ray images, the increase in pre - training scale shows a significant performance improvement in full - sample transfer learning, especially on larger medical X - ray target data sets. However, in smaller target data sets or few - sample transfer learning, this improvement is not obvious. 3. **Advantages of large - scale natural image pre - training**: - Large - scale natural image pre - training (such as using ImageNet - 21k) can even be superior to models pre - trained with large - scale medical X - ray data in full - sample transfer learning, especially on larger medical X - ray target data sets. This indicates that by increasing the scale of pre - training models and general natural image data, high - quality cross - domain transfer performance can be obtained without relying on a large amount of domain - specific data. ### Conclusion The authors conclude that significantly increasing the scale of pre - training models and general natural image data can achieve high - quality cross - domain transfer learning, thereby reducing the dependence on a large amount of domain - specific data. This finding is of great significance for practical applications, especially in the field of medical imaging, where it is often very difficult to obtain a large amount of labeled data. Through this study, the authors provide new insights into understanding the influence of pre - training scale on the effect of transfer learning and demonstrate the potential of large - scale pre - training in cross - domain tasks.

Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images

Supervised Transfer Learning at Scale for Medical Imaging

Targeted transfer learning to improve performance in small medical physics datasets

Why does my medical AI look at pictures of birds? Exploring the efficacy of transfer learning across domain boundaries

Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images

Transfusion: Understanding Transfer Learning for Medical Imaging

Effect of Training Data Volume on Performance of Convolutional Neural Network Pneumothorax Classifiers

CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation

Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging

Transfer learning from a sparsely annotated dataset of 3D medical images

Is in-domain data beneficial in transfer learning for landmarks detection in x-ray images?

How to Train Your CheXDragon: Training Chest X-Ray Models for Transfer to Novel Tasks and Healthcare Systems

Lightweight multi-scale classification of chest radiographs via size-specific batch normalization

A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis

Evaluating Knowledge Transfer in Neural Network for Medical Images

The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Images

What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors

Analyzing Transfer Learning of Vision Transformers for Interpreting Chest Radiography

Video Pretraining Advances 3D Deep Learning on Chest CT Tasks

Pre-training on High Definition X-ray Images: An Experimental Study

Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification