Abstract:Given a set of pre-trained models, how can we quickly and accurately find the most useful pre-trained model for a downstream task? Transferability measurement is to quantify how transferable is a pre-trained model learned on a source task to a target task. It is used for quickly ranking pre-trained models for a given task and thus becomes a crucial step for transfer learning. Existing methods measure transferability as the discrimination ability of a source model for a target data before transfer learning, which cannot accurately estimate the fine-tuning performance. Some of them restrict the application of transferability measurement in selecting the best supervised pre-trained models that have classifiers. It is important to have a general method for measuring transferability that can be applied in a variety of situations, such as selecting the best self-supervised pre-trained models that do not have classifiers, and selecting the best transferring layer for a target task. In this work, we propose TMI (TRANSFERABILITY MEASUREMENT WITH INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure transferability. We view transferability as the generalization of a pre-trained model on a target task by measuring intra-class feature variance. Intra-class variance evaluates the adaptability of the model to a new task, which measures how transferable the model is. Compared to previous studies that estimate how discriminative the models are, intra-class variance is more accurate than those as it does not require an optimal feature extractor and classifier. Extensive experiments on real-world datasets show that TMI outperforms competitors for selecting the top-5 best models, and exhibits consistently better correlation in 13 out of 17 cases.

Which Model to Transfer? A Survey on Transferability Estimation

A Comprehensive Survey on Transfer Learning

How to Determine the Most Powerful Pre-trained Language Model Without Brute Force Fine-tuning? an Empirical Survey

Transferability in Deep Learning: A Survey

Deep Model Transferability from Attribution Maps

A Survey on Heterogeneous Transfer Learning

Understanding the Transferability of Representations via Task-Relatedness

Transfer Learning and Its Application Research

Pick the Best Pre-trained Model: Towards Transferability Estimation for Medical Image Segmentation

An Information-Theoretic Approach to Transferability in Task Transfer Learning

Frustratingly Easy Transferability Estimation

Efficient Transferability Assessment for Selection of Pre-trained Detectors

Transfer Learning for Cross-Dataset Recognition: A Survey

To transfer or not transfer: Unified transferability metric and analysis

Exploring and Predicting Transferability across NLP Tasks

A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning

A Recent Survey of Heterogeneous Transfer Learning

Quantifying and Improving Transferability in Domain Generalization

Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance

On the Transferability of Learning Models for Semantic Segmentation for Remote Sensing Data

Where and How to Transfer: Knowledge Aggregation-Induced Transferability Perception for Unsupervised Domain Adaptation.