Abstract:Transformers have revolutionized the object detection landscape by introducing DETRs, acclaimed for their simplicity and efficacy. Despite their advantages, the substantial size of these models poses significant challenges for practical deployment, particularly in resource-constrained environments. This paper addresses the challenge of compressing DETR by leveraging knowledge distillation, a technique that holds promise for maintaining model performance while reducing size. A critical aspect of DETRs' performance is their reliance on queries to interpret object representations accurately. Traditional distillation methods often focus exclusively on positive queries, identified through bipartite matching, neglecting the rich information present in hard-negative queries. Our visual analysis indicates that hard-negative queries, focusing on foreground elements, are crucial for enhancing distillation outcomes. To this end, we introduce a novel Group Query Selection strategy, which diverges from traditional query selection in DETR distillation by segmenting queries based on their Generalized Intersection over Union (GIoU) with ground truth objects, thereby uncovering valuable hard-negative queries for distillation. Furthermore, we present the Knowledge Distillation via Query Selection for DETR (QSKD) framework, which incorporates Attention-Guided Feature Distillation (AGFD) and Local Alignment Prediction Distillation (LAPD). These components optimize the distillation process by focusing on the most informative aspects of the teacher model's intermediate features and output. Our comprehensive experimental evaluation of the MS-COCO dataset demonstrates the effectiveness of our approach, significantly improving average precision (AP) across various DETR architectures without incurring substantial computational costs. Specifically, the AP of Conditional DETR ResNet-18 increased from 35.8 to 39.9.

HA-DQS-Net: dynamic query design based on transformer with hollow attention

Anchor DETR: Query Design for Transformer-Based Detector

Guiding Query Position and Performing Similar Attention for Transformer-Based Detection Heads

Dynamic Cascade Query Selection for Oriented Object Detection

Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement

D2Q-DETR: Decoupling and Dynamic Queries for Oriented Object Detection with Transformers

QETR: A Query-Enhanced Transformer for Remote Sensing Image Object Detection

Dynamic multi-headed self-attention and multiscale enhancement vision transformer for object detection

DQ-DETR: DETR with Dynamic Query for Tiny Object Detection

Long-short Range Adaptive Transformer with Dynamic Sampling for 3D Object Detection

DHS-DETR: Efficient DETRs with Dynamic Head Switching

EMO2-DETR: Efficient-Matching Oriented Object Detection with Transformers.

Knowledge Distillation via Query Selection for Detection Transformer

DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion

Category-Aware Transformer Network for Better Human-Object Interaction Detection

Dynamic Object Queries for Transformer-based Incremental Object Detection

DetectorNet

DECO: Query-Based End-to-End Object Detection with ConvNets

Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection

DETR-ORD: An Improved DETR Detector for Oriented Remote Sensing Object Detection with Feature Reconstruction and Dynamic Query