Abstract:Transfer learning enables to re-use knowledge learned on a source task to help learning a target task. A simple form of transfer learning is common in current state-of-the-art computer vision models, i.e. pre-training a model for image classification on the ILSVRC dataset, and then fine-tune on any target task. However, previous systematic studies of transfer learning have been limited and the circumstances in which it is expected to work are not fully understood. In this paper we carry out an extensive experimental exploration of transfer learning across vastly different image domains (consumer photos, autonomous driving, aerial imagery, underwater, indoor scenes, synthetic, close-ups) and task types (semantic segmentation, object detection, depth estimation, keypoint detection). Importantly, these are all complex, structured output tasks types relevant to modern computer vision applications. In total we carry out over 2000 transfer learning experiments, including many where the source and target come from different image domains, task types, or both. We systematically analyze these experiments to understand the impact of image domain, task type, and dataset size on transfer learning performance. Our study leads to several insights and concrete recommendations: (1) for most tasks there exists a source which significantly outperforms ILSVRC'12 pre-training; (2) the image domain is the most important factor for achieving positive transfer; (3) the source dataset should \emph{include} the image domain of the target dataset to achieve best results; (4) at the same time, we observe only small negative effects when the image domain of the source task is much broader than that of the target; (5) transfer across task types can be beneficial, but its success is heavily dependent on both the source and target task types.

EigenTransfer

Deep Model Transferability from Attribution Maps

A Visual Analytical Approach for Transfer Learning in Classification.

Ensemble Transfer Learning Algorithm

Understanding How Feature Structure Transfers in Transfer Learning.

Unsupervisedly Learned Latent Graphs as Transferable Representations

A General Transitive Transfer Learning Framework for Cross-Optical Sensor Remote Sensing Image Scene Understanding

Feasibility of Transfer Learning: A Mathematical Framework

Spectral Domain-Transfer Learning

HetEOTL: an Algorithm for Heterogeneous Online Transfer Learning

Constrained Deep Transfer Feature Learning and its Applications

Heterogeneous Transfer Learning for Image Classification.

A new semi-supervised inductive transfer learning framework: Co-Transfer

Cross-Domain Kernel Induction for Transfer Learning

Deep Transfer Learning: Model Framework and Error Analysis

Understanding the Transferability of Representations via Task-Relatedness

A Mathematical Framework for Quantifying Transferability in Multi-source Transfer Learning

Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries

GLoMo: Unsupervised Learning of Transferable Relational Graphs

Flexible deep transfer learning by separate feature embeddings and manifold alignment

Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types