Abstract:The recently proposed DEtection TRansformer (DETR) has established a fully end-to-end paradigm for object detection. However, DETR suffers from slow training convergence, which hinders its applicability to various detection tasks. We observe that DETR's slow convergence is largely attributed to the difficulty in matching object queries to relevant regions due to the unaligned semantics between object queries and encoded image features. With this observation, we design Semantic-Aligned-Matching DETR++ (SAM-DETR++) to accelerate DETR's convergence and improve detection performance. The core of SAM-DETR++ is a plug-and-play module that projects object queries and encoded image features into the same feature embedding space, where each object query can be easily matched to relevant regions with similar semantics. Besides, SAM-DETR++ searches for multiple representative keypoints and exploits their features for semantic-aligned matching with enhanced representation capacity. Furthermore, SAM-DETR++ can effectively fuse multi-scale features in a coarse-to-fine manner on the basis of the designed semantic-aligned matching. Extensive experiments show that the proposed SAM-DETR++ achieves superior convergence speed and competitive detection accuracy. Additionally, as a plug-and-play method, SAM-DETR++ can complement existing DETR convergence solutions with even better performance, achieving 44.8% AP with merely 12 training epochs and 49.1% AP with 50 training epochs on COCO val 2017 with ResNet-50. Codes are available at https://github.com/ZhangGongjie/SAM-DETR .

Detrex: Benchmarking Detection Transformers

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Towards Data-Efficient Detection Transformers

DETR-ORD: An Improved DETR Detector for Oriented Remote Sensing Object Detection with Feature Reconstruction and Dynamic Query

DETR++: Taming Your Multi-Scale Detection Transformer

L-DETR: A Light-Weight Detector for End-to-End Object Detection With Transformers

Detection Transformer with Stable Matching

Rank-DETR for High Quality Object Detection

Investigating the Robustness and Properties of Detection Transformers (DETR) Toward Difficult Images

Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR

AugDETR: Improving Multi-scale Learning for Detection Transformer

Team DETR: Guide Queries As a Professional Team in Detection Transformers

RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer

End-to-End Object Detection with Adaptive Clustering Transformer

Semantic-Aligned Matching for Enhanced DETR Convergence and Multi-Scale Feature Fusion

IST-DETR: Improved DETR for Infrared Small Target Detection

DHS-DETR: Efficient DETRs with Dynamic Head Switching

DETR Does Not Need Multi-Scale or Locality Design

Cascade-DETR: Delving into High-Quality Universal Object Detection

EF-DETR: A Lightweight Transformer-Based Object Detector with an Encoder-Free Neck

Semi-DETR: Semi-Supervised Object Detection with Detection Transformers