Abstract:We address the challenge of getting efficient yet accurate recognition systems with limited labels. While recognition models improve with model size and amount of data, many specialized applications of computer vision have severe resource constraints both during training and inference. Transfer learning is an effective solution for training with few labels, however often at the expense of a computationally costly fine-tuning of large base models. We propose to mitigate this unpleasant trade-off between compute and accuracy via semi-supervised cross-domain distillation from a set of diverse source models. Initially, we show how to use task similarity metrics to select a single suitable source model to distill from, and that a good selection process is imperative for good downstream performance of a target model. We dub this approach DistillNearest. Though effective, DistillNearest assumes a single source model matches the target task, which is not always the case. To alleviate this, we propose a weighted multi-source distillation method to distill multiple source models trained on different domains weighted by their relevance for the target task into a single efficient model (named DistillWeighted). Our methods need no access to source data, and merely need features and pseudo-labels of the source models. When the goal is accurate recognition under computational constraints, both DistillNearest and DistillWeighted approaches outperform both transfer learning from strong ImageNet initializations as well as state-of-the-art semi-supervised techniques such as FixMatch. Averaged over 8 diverse target tasks our multi-source method outperforms the baselines by 5.6%-points and 4.5%-points, respectively.

Leveraging knowledge distillation for partial multi-task learning from multiple remote sensing datasets

Data exploitation: multi-task learning of object detection and semantic segmentation on partially annotated data

Knowledge Distillation for Object Detection: from generic to remote sensing datasets

Learning Multiple Dense Prediction Tasks from Partially Annotated Data

Two-Way Assistant: A Knowledge Distillation Object Detection Method for Remote Sensing Images

Expanding Deep Learning-based Sensing Systems with Multi-Source Knowledge Transfer

Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation

Unpaired Multi-modal Segmentation via Knowledge Distillation

MSTNet-KD: Multilevel Transfer Networks Using Knowledge Distillation for the Dense Prediction of Remote-Sensing Images

Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation

Multi-Knowledge Aggregation and Transfer for Semantic Segmentation

Class-Incremental Learning for Remote Sensing Images Based on Knowledge Distillation

Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning

Multi-instance semantic similarity transferring for knowledge distillation

Target Detection Model Distillation Using Feature Transition and Label Registration for Remote Sensing Imagery

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images

Centerness-based Instance-aware Knowledge Distillation with Task-wise Mutual Lifting for Object Detection on Drone Imagery

Multimodal Knowledge Distillation for Arbitrary-Oriented Object Detection in Aerial Images

Distilling from Similar Tasks for Transfer Learning on a Budget

Task Integration Distillation for Object Detectors

Joint multi-modal Self-Supervised pre-training in Remote Sensing: Application to Methane Source Classification