Abstract:Neural radiance fields (NeRF) and its subsequent variants have led to remarkable progress in neural rendering. While most of recent neural rendering works focus on objects and small-scale scenes, developing neural rendering methods for city-scale scenes is of great potential in many real-world applications. However, this line of research is impeded by the absence of a comprehensive and high-quality dataset, yet collecting such a dataset over real city-scale scenes is costly, sensitive, and technically difficult. To this end, we build a large-scale, comprehensive, and high-quality synthetic dataset for city-scale neural rendering researches. Leveraging the Unreal Engine 5 City Sample project, we develop a pipeline to easily collect aerial and street city views, accompanied by ground-truth camera poses and a range of additional data modalities. Flexible controls over environmental factors like light, weather, human and car crowd are also available in our pipeline, supporting the need of various tasks covering city-scale neural rendering and beyond. The resulting pilot dataset, MatrixCity, contains 67k aerial images and 452k street images from two city maps of total size $28km^2$. On top of MatrixCity, a thorough benchmark is also conducted, which not only reveals unique challenges of the task of city-scale neural rendering, but also highlights potential improvements for future works. The dataset and code will be publicly available at our project page: <a class="link-external link-https" href="https://city-super.github.io/matrixcity/" rel="external noopener nofollow">this https URL</a>.

The Cityscapes Dataset

SkyScapes -- Fine-Grained Semantic Understanding of Aerial Scenes

SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds

Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset

Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges

CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data

OmniCity: Omnipotent City Understanding with Multi-level and Multi-view Images

Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection

SUM: A Benchmark Dataset of Semantic Urban Meshes

Semantic segmentation of urban environments: Leveraging U-Net deep learning model for cityscape image analysis

MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond

A New Urban Objects Detection Framework Using Weakly Annotated Sets

CityPulse: Fine-Grained Assessment of Urban Change with Street View Time Series

The ParallelEye Dataset: Constructing Large-Scale Artificial Scenes for Traffic Vision Research

Semantic Understanding of Scenes Through the ADE20K Dataset

The Fieldscapes Dataset for Semantic Field Scene Understanding

Urban Mosaic: Visual Exploration of Streetscapes Using Large-Scale Image Data

A Fine-Grained Dataset and its Efficient Semantic Segmentation for Unstructured Driving Scenarios

The EuroCity Persons Dataset: A Novel Benchmark for Object Detection

DublinCity: Annotated LiDAR Point Cloud and its Applications

RoScenes: A Large-scale Multi-view 3D Dataset for Roadside Perception