Abstract:Object detection is a fundamental task in computer vision. Over the past several years, convolutional neural network (CNN)-based object detection models have significantly improved detection accuracyin terms of average precision (AP). Furthermore, feature pyramid networks (FPNs) are essential modules for object detection models to consider various object scales. However, the AP for small objects is lower than the AP for medium and large objects. It is difficult to recognize small objects because they do not have sufficient information, and information is lost in deeper CNN layers. This paper proposes a new FPN model named ssFPN (scale sequence (S2) feature-based feature pyramid network) to detect multi-scale objects, especially small objects. We propose a new scale sequence (S2) feature that is extracted by 3D convolution on the level of the FPN. It is defined and extracted from the FPN to strengthen the information on small objects based on scale-space theory. Motivated by this theory, the FPN is regarded as a scale space and extracts a scale sequence (S2) feature by three-dimensional convolution on the level axis of the FPN. The defined feature is basically scale-invariant and is built on a high-resolution pyramid feature map for small objects. Additionally, the deigned S2 feature can be extended to most object detection models based on FPNs. We also designed a feature-level super-resolution approach to show the efficiency of the scale sequence (S2) feature. We verified that the scale sequence (S2) feature could improve the classification accuracy for low-resolution images by training a feature-level super-resolution model. To demonstrate the effect of the scale sequence (S2) feature, experiments on the scale sequence (S2) feature built-in object detection approach including both one-stage and two-stage models were conducted on the MS COCO dataset. For the two-stage object detection models Faster R-CNN and Mask R-CNN with the S2 feature, AP improvements of up to 1.6% and 1.4%, respectively, were achieved. Additionally, the APS of each model was improved by 1.2% and 1.1%, respectively. Furthermore, the one-stage object detection models in the YOLO series were improved. For YOLOv4-P5, YOLOv4-P6, YOLOR-P6, YOLOR-W6, and YOLOR-D6 with the S2 feature, 0.9%, 0.5%, 0.5%, 0.1%, and 0.1% AP improvements were observed. For small object detection, the APS increased by 1.1%, 1.1%, 0.9%, 0.4%, and 0.1%, respectively. Experiments using the feature-level super-resolution approach with the proposed scale sequence (S2) feature were conducted on the CIFAR-100 dataset. By training the feature-level super-resolution model, we verified that ResNet-101 with the S2 feature trained on LR images achieved a 55.2% classification accuracy, which was 1.6% higher than for ResNet-101 trained on HR images.

A Method for Training Object Scale Estimation System using Feature Extraction Enhancement with Depth Estimation

Image Segmentation and Scale-Invariable Feature Extraction for Level Bars Calibration

An Analysis of Scale Invariance in Object Detection - SNIP

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

SelfTune: Metrically Scaled Monocular Depth Estimation through Self-Supervised Learning

ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond

ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation

Scale Enhancement Network for Object Detection in Aerial Images

Addressing the Scale Shrinkage Problem in Learning-based Binocular Depth Estimation

Variable Scale Learning for Visual Object Tracking

Deep Scale Feature For Visual Tracking

STEERER: Resolving Scale Variations for Counting and Localization via Selective Inheritance Learning

Scale Equalization for Multi-Level Feature Fusion

ssFPN: Scale Sequence (S2) Feature-Based Feature Pyramid Network for Object Detection

RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery

Object Segmentation Ensuring Consistency Across Multi-Viewpoint Images

Delving into the Scale Variance Problem in Object Detection

ScaleDet: A Scalable Multi-Dataset Object Detector

SAN: Learning Relationship between Convolutional Features for Multi-Scale Object Detection

SizeNet: Object Recognition via Object Real Size-based Convolutional Networks

Exploiting Cross-scale Consistency for Object Detection in Aerial Images