Abstract:Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other -- independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such as separate test sets, e.g. due to data privacy), we investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation -- a task made particularly difficult as additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic to pretrained model pairings would unlock auxiliary gains and knowledge fusion from any model repository without restrictions on model and problem specifics - including from weaker, lower-performance models. This work therefore provides an initial, in-depth exploration on the viability of such general-purpose knowledge transfer. Across large-scale experiments, we first reveal the shortcomings of standard knowledge distillation techniques, and then propose a much more general extension through data partitioning for successful transfer between nearly all pretrained models, which we show can also be done unsupervised. Finally, we assess both the scalability and impact of fundamental model properties on successful model-agnostic knowledge transfer.

Learning Deep Representations with Probabilistic Knowledge Transfer

Collaborative Knowledge Distillation Via Multiknowledge Transfer.

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Variational Information Distillation for Knowledge Transfer

Adaptive Explicit Knowledge Transfer for Knowledge Distillation

Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network

Follow Your Path: A Progressive Method for Knowledge Distillation

Knowledge Transfer Between Artificial Intelligence Systems

Harmonizing knowledge Transfer in Neural Network with Unified Distillation

Improving Knowledge Distillation With a Customized Teacher

A Probabilistic Approach to Knowledge Translation

Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge Graphs

Knowledge Transfer via Dense Cross-Layer Mutual-Distillation

Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

Expanding Deep Learning-based Sensing Systems with Multi-Source Knowledge Transfer

Analysis of Knowledge Transfer in Kernel Regime

Robust Knowledge Transfer Via Hybrid Forward on the Teacher-Student Model

Knowledge Projection for Deep Neural Networks

Learning to Project for Cross-Task Knowledge Distillation

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model