Abstract:Multi-scale image resolution is a de facto standard approach in modern object detectors, such as DETR. This technique allows for the acquisition of various scale information from multiple image resolutions. However, manual hyperparameter selection of the resolution can restrict its flexibility, which is informed by prior knowledge, necessitating human intervention. This work introduces a novel strategy for learnable resolution, called Elastic-DETR, enabling elastic utilization of multiple image resolutions. Our network provides an adaptive scale factor based on the content of the image with a compact scale prediction module (< 2 GFLOPs). The key aspect of our method lies in how to determine the resolution without prior knowledge. We present two loss functions derived from identified key components for resolution optimization: scale loss, which increases adaptiveness according to the image, and distribution loss, which determines the overall degree of scaling based on network performance. By leveraging the resolution's flexibility, we can demonstrate various models that exhibit varying trade-offs between accuracy and computational complexity. We empirically show that our scheme can unleash the potential of a wide spectrum of image resolutions without constraining flexibility. Our models on MS COCO establish a maximum accuracy gain of 3.5%p or 26% decrease in computation than MS-trained DN-DETR.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in modern object detectors, although multi - scale image resolutions can obtain information at different scales, they rely on manually - selected hyper - parameters, which limits flexibility and requires human intervention. Therefore, the authors propose a new strategy - Elastic - DETR - to make image resolutions learnable, enabling the network to adaptively adjust the resolution according to the image content, thereby improving the flexibility and efficiency of the model. Specifically, the main contributions of the paper include: 1. **Proposing a learnable image resolution optimization strategy**: Elastic - DETR can elastically utilize multiple image resolutions, providing a general solution to optimize hyper - parameters without relying on prior knowledge. 2. **Constructing a compact architecture for predicting scaling factors**: This architecture can perform adaptive prediction according to the image content, making the scaling factors targeted. 3. **Introducing two new loss functions**: These loss functions are based on key components extracted from human behavior characteristics and can optimize scaling factors without relying on prior knowledge. Specifically: - **Scale Loss**: Optimize the scaling factor according to the object size to enhance the adaptability to objects of different scales. - **Distribution Loss**: Optimize the overall distribution of the scaling factor according to the network performance to ensure a reasonable range of the scaling factor. 4. **Making the first attempt to optimize image resolution in the DETR - based network**: Through experimental verification, this method can achieve an accuracy improvement of up to 3.5% or a 26% reduction in the amount of computation on the MS COCO dataset. Through these improvements, Elastic - DETR not only improves the flexibility and efficiency of the model but also performs well in handling small objects, effectively alleviating the limitations in traditional methods.

Elastic-DETR: Making Image Resolution Learnable with Content-Specific Network Prediction

Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity

Cross Resolution Encoding-Decoding For Detection Transformers

DETRs with Collaborative Hybrid Assignments Training

Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR

Deformable DETR: Deformable Transformers for End-to-End Object Detection

DETR++: Taming Your Multi-Scale Detection Transformer

DETR Doesn't Need Multi-Scale or Locality Design

Identifying Reliable Predictions in Detection Transformers

DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

Semantic-Aligned Matching for Enhanced DETR Convergence and Multi-Scale Feature Fusion

Two-Stage Multi-Scale Resolution-Adaptive Network for Low-Resolution Face Recognition

Efficient DETR: Improving End-to-End Object Detector with Dense Prior

PR-Deformable DETR: DETR for Remote Sensing Object Detection

PMG-DETR: fast convergence of DETR with position-sensitive multi-scale attention and grouped queries

RestoreDet: Degradation Equivariant Representation for Object Detection in Low Resolution Images

Small Object Detection by DETR via Information Augmentation and Adaptive Feature Fusion

Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection

Cross-Domain Knowledge Distillation for Low-Resolution Human Pose Estimation

Drone-DETR: Efficient Small Object Detection for Remote Sensing Image Using Enhanced RT-DETR Model

EfficientDet: Scalable and Efficient Object Detection