Abstract:Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale multi-modal (LiDAR + RGB) data. Surprisingly, although semantic class labels naturally follow a long-tailed distribution, existing benchmarks only focus on a few common classes (e.g., pedestrian and car) and neglect many rare but crucial classes (e.g., emergency vehicle and stroller). However, AVs must reliably detect both common and rare classes for safe operation in the open world. We address this challenge by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates all annotated classes, including those in-the-tail. We address LT3D with hierarchical losses that promote feature sharing across classes, and introduce diagnostic metrics that award partial credit to ``reasonable'' mistakes with respect to the semantic hierarchy (e.g., mistaking a child for an adult). Further, we point out that rare-class accuracy is particularly improved via multi-modal late fusion (MMLF) of independently trained uni-modal LiDAR and RGB detectors. Importantly, such an MMLF framework allows us to leverage large-scale uni-modal datasets (with more examples for rare classes) to train better uni-modal detectors, unlike prevailing end-to-end trained multi-modal detectors that require paired multi-modal data. Finally, we examine three critical components of our simple MMLF approach from first principles and investigate whether to train 2D or 3D RGB detectors for fusion, whether to match RGB and LiDAR detections in 3D or the projected 2D image plane, and how to fuse matched detections. Our proposed MMLF approach significantly improves LT3D performance over prior work, particularly improving rare class performance from 12.8 to 20.0 mAP!

MT-Net Submission to the Waymo 3D Detection Leaderboard

MT-SSD: Single-Stage 3D Object Detector Based on Magnification Transformation

CenterAtt: Fast 2-stage Center Attention Network

1st Place Solution for Waymo Open Dataset Challenge -- 3D Detection and Domain Adaptation

LidarMultiNet: Unifying LiDAR Semantic Segmentation, 3D Object Detection, and Panoptic Segmentation in a Single Multi-task Network

2nd Place Solution for Waymo Open Dataset Challenge -- Real-time 2D Object Detection

1st Place Solutions for Waymo Open Dataset Challenges -- 2D and 3D Tracking

A Multi-view 3D Vehicle Detection Method Based On Novel 3D Proposal Generation Method

Point Transformer V3 Extreme: 1st Place Solution for 2024 Waymo Open Dataset Challenge in Semantic Segmentation

MV-FCOS3D++: Multi-View Camera-Only 4D Object Detection with Pretrained Monocular Backbones

Multi-View 3D Object Detection Network for Autonomous Driving

DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds

1st Place Solutions of Waymo Open Dataset Challenge 2020 -- 2D Object Detection Track

Long-Tailed 3D Detection via Multi-Modal Fusion

MSPV3D: Multi-Scale Point-Voxels 3D Object Detection Net

PVGNet: A Bottom-Up One-Stage 3D Object Detector with Integrated Multi-Level Features

Multimodal Virtual Point 3D Detection

The 2nd Place Solution for 2023 Waymo Open Sim Agents Challenge

Ground-aware Monocular 3D Object Detection for Autonomous Driving

PA3DNet: 3-D Vehicle Detection with Pseudo Shape Segmentation and Adaptive Camera-LiDAR Fusion