Abstract:Balancing the trade-off between accuracy and speed for obtaining higher performance without sacrificing the inference time is a challenging topic for object detection task. Knowledge distillation, which serves as a kind of model compression techniques, provides a potential and feasible way to handle above efficiency and effectiveness issue through transferring the dark knowledge from the sophisticated teacher detector to the simple student one. Despite demonstrating promising solutions to make harmonies between accuracy and speed, current knowledge distillation for object detection methods still suffer from two limitations. Firstly, most of the methods are inherited or refereed from the frameworks in image classification task, and deploy an implicit manner by imitating or constraining the features from the intermediate layers or the output predictions between the teacher and student models. While little consideration has been raised to the intrinsic relevance of the classification and localization predictions in object detection task. Besides, these methods fail to investigate the relationship between detection and distillation tasks in knowledge distillation pipeline, and they train the whole network by simply integrating losses from these two different tasks through hand-crafted designation parameters. For addressing the aforementioned issues, we propose a novel Relation Knowledge Distillation by Auxiliary Learning for Object Detection (ReAL) method in this paper. Specifically, we first design a prediction relation distillation module which makes the student model directly mimic the output predictions from the teacher one, and conduct self and mutual relation distillation losses to excavate the relation information between teacher and student models. Moreover, for better devolving into the relationship between different tasks in distillation pipeline, we introduce the auxiliary learning into knowledge distillation for object detection and develop a dynamic weight adaptation strategy. Through regarding detection task as primary task and treating distillation task as auxiliary task in auxiliary learning framework, we dynamically adjust and regularize the corresponding weights of the losses for these tasks during the training process. Experiments on MS COCO dataset are conducted using various detector combinations of teacher and student models and the results show that our proposed ReAL can achieve obvious improvement on different distillation model configurations, while performing favorably against state-of-the-arts.

Adversarial Multi-Teacher Distillation for Semi-Supervised Relation Extraction.

MKDAT: Multi-Level Knowledge Distillation with Adaptive Temperature for Distantly Supervised Relation Extraction

MiDTD: A Simple and Effective Distillation Framework for Distantly Supervised Relation Extraction

A Multi-teacher Knowledge Distillation Framework for Distantly Supervised Relation Extraction with Flexible Temperature

Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training

Transfer Learning for Relation Extraction Via Relation-Gated Adversarial Learning

Adversarial Training for Supervised Relation Extraction

Semi-Supervised Dual Relation Learning for Multi-Label Classification

Distantly Supervised Relation Extraction Based on Residual Attention and Self Learning

A Review on Semi-Supervised Relation Extraction

Interactive Lexical and Semantic Graphs for Semisupervised Relation Extraction.

Semi-supervised Relation Extraction via Data Augmentation and Consistency-training

A Relationship Extraction Model Based on Reinforcement Learning and Multi-Teacher Knowledge Distillation

Relation-Based Multi-Teacher Knowledge Distillation

Semi-supervised Relation Extraction Via Incremental Meta Self-Training

Relation Knowledge Distillation by Auxiliary Learning for Object Detection

Competition or Cooperation? Exploring Unlabeled Data Via Challenging Minimax Game for Semi-supervised Relation Extraction.

Noise-Robust Semi-Supervised Learning for Distantly Supervised Relation Extraction

Adversarial Learning for Supervised and Semi-supervised Relation Extraction in Biomedical Literature

Reducing Wrong Labels for Distantly Supervised Relation Extraction With Reinforcement Learning

Multi-Instance Active Learning for Relation Extraction