Abstract:Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale multi-modal (LiDAR + RGB) data. Surprisingly, although semantic class labels naturally follow a long-tailed distribution, existing benchmarks only focus on a few common classes (e.g., pedestrian and car) and neglect many rare but crucial classes (e.g., emergency vehicle and stroller). However, AVs must reliably detect both common and rare classes for safe operation in the open world. We address this challenge by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates all annotated classes, including those in-the-tail. We address LT3D with hierarchical losses that promote feature sharing across classes, and introduce diagnostic metrics that award partial credit to ``reasonable'' mistakes with respect to the semantic hierarchy (e.g., mistaking a child for an adult). Further, we point out that rare-class accuracy is particularly improved via multi-modal late fusion (MMLF) of independently trained uni-modal LiDAR and RGB detectors. Importantly, such an MMLF framework allows us to leverage large-scale uni-modal datasets (with more examples for rare classes) to train better uni-modal detectors, unlike prevailing end-to-end trained multi-modal detectors that require paired multi-modal data. Finally, we examine three critical components of our simple MMLF approach from first principles and investigate whether to train 2D or 3D RGB detectors for fusion, whether to match RGB and LiDAR detections in 3D or the projected 2D image plane, and how to fuse matched detections. Our proposed MMLF approach significantly improves LT3D performance over prior work, particularly improving rare class performance from 12.8 to 20.0 mAP!

Class-quantity and class-difficulty based methods for long-tailed road marking detection

An Efficient Framework for Detection and Recognition of Numerical Traffic Signs

Foreground Gating and Background Refining Network for Surveillance Object Detection

Class-Guided Triple Head Prediction Network for Long-Tail Object Detection

Towards Resolving the Challenge of Long-tail Distribution in UAV Images for Object Detection

Multiclass objects detection algorithm using DarkNet-53 and DenseNet for intelligent vehicles

A unified and costless approach for improving small and long-tail object detection in aerial images of traffic scenarios

Road Traffic Sign Detection Method Based on RTS R-CNN Instance Segmentation Network

A two-stage real-time YOLOv2-based road marking detector with lightweight spatial transformation-invariant classification

Tail Classes Matter: Long-Tailed Object Detection Revisited.

Traffic Signal Detection and Classification in Street Views Using an Attention Model

Towards Long-Tailed 3D Detection

Long-Tailed Object Detection for Multimodal Remote Sensing Images

TANet: Robust 3D Object Detection from Point Clouds with Triple Attention

An advanced YOLOv3 method for small-scale road object detection

Overcoming the Challenges of Long-tail Distribution in Night-time Vehicle Detection

Long-Tailed 3D Detection via Multi-Modal Fusion

Traffic Sign Recognition With Lightweight Two-Stage Model in Complex Scenes

Rethinking the Detection Head Configuration for Traffic Object Detection

Traffic-Sign Detection and Classification in the Wild

A novel real-time object detection method for complex road scenes based on YOLOv7-tiny