Abstract:Autonomous driving has attracted tremendous attention especially in the past few years. The key techniques for a self-driving car include solving tasks like 3D map construction, self-localization, parsing the driving road and understanding objects, which enable vehicles to reason and act. However, large scale data set for training and system evaluation is still a bottleneck for developing robust perception models. In this paper, we present the ApolloScape dataset [1] and its applications for autonomous driving. Compared with existing public datasets from real scenes, e.g. KITTI [2] or Cityscapes [3], ApolloScape contains much large and richer labelling including holistic semantic dense point cloud for each site, stereo, per-pixel semantic labelling, lanemark labelling, instance segmentation, 3D car instance, high accurate location for every frame in various driving videos from multiple sites, cities and daytimes. For each task, it contains at lease 15x larger amount of images than SOTA datasets. To label such a complete dataset, we develop various tools and algorithms specified for each task to accelerate the labelling process, such as 3D-2D segment labeling tools, active labelling in videos etc. Depend on ApolloScape, we are able to develop algorithms jointly consider the learning and inference of multiple tasks. In this paper, we provide a sensor fusion scheme integrating camera videos, consumer-grade motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robust self-localization and semantic segmentation for autonomous driving. We show that practically, sensor fusion and joint learning of multiple tasks are beneficial to achieve a more robust and accurate system. We expect our dataset and proposed relevant algorithms can support and motivate researchers for further development of multi-sensor fusion and multi-task learning in the field of computer vision.

InfraParis: A multi-modal and multi-task autonomous driving dataset

aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception

OpenMPD: An Open Multimodal Perception Dataset for Autonomous Driving

Road and Railway Smart Mobility: A High-Definition Ground Truth Hybrid Dataset

ParisLuco3D: A high-quality target dataset for domain generalization of LiDAR perception

EnvoDat: A Large-Scale Multisensory Dataset for Robotic Spatial Awareness and Semantic Reasoning in Heterogeneous Environments

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

AmodalSynthDrive: A Synthetic Amodal Perception Dataset for Autonomous Driving

nuScenes: A Multimodal Dataset for Autonomous Driving

OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving

Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping

KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors

ROAD: The ROad event Awareness Dataset for Autonomous Driving

DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection

The ApolloScape Open Dataset for Autonomous Driving and its Application

LiDAR-CS Dataset: LiDAR Point Cloud Dataset with Cross-Sensors for 3D Object Detection

Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks

FRSign: A Large-Scale Traffic Light Dataset for Autonomous Trains

Scalability in Perception for Autonomous Driving: Waymo Open Dataset.