Abstract:With the recent rise of Large Language Models (LLMs), Vision-Language Models (VLMs), and other general foundation models, there is growing potential for multimodal, multi-task embodied agents that can operate in diverse environments given only natural language as input. One such application area is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the spatial reasoning and semantic understanding required, particularly in arbitrary scenes that may contain many objects belonging to fine-grained classes. To address this challenge, we curate the largest real-world dataset for Vision and Language-guided Action in 3D Scenes (VLA-3D), consisting of over 11.5K scanned 3D indoor rooms from existing datasets, 23.5M heuristically generated semantic relations between objects, and 9.7M synthetically generated referential statements. Our dataset consists of processed 3D point clouds, semantic object and room annotations, scene graphs, navigable free space annotations, and referential language statements that specifically focus on view-independent spatial relations for disambiguating objects. The goal of these features is to aid the downstream task of navigation, especially on real-world systems where some level of robustness must be guaranteed in an open world of changing scenes and imperfect language. We benchmark our dataset with current state-of-the-art models to obtain a performance baseline. All code to generate and visualize the dataset is publicly released, see <a class="link-external link-https" href="https://github.com/HaochenZ11/VLA-3D" rel="external noopener nofollow">this https URL</a>. With the release of this dataset, we hope to provide a resource for progress in semantic 3D scene understanding that is robust to changes and one which will aid the development of interactive indoor navigation systems.

Large-Scale Indoor Visual-Geometric Multimodal Dataset and Benchmark for Novel View Synthesis

Benchmarking Large-Scale Multi-View 3D Reconstruction Using Realistic Synthetic Images

InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

Scalable Neural Indoor Scene Rendering.

Scalable Indoor Novel-View Synthesis using Drone-Captured 360 Imagery with 3D Gaussian Splatting

MINERVAS: Massive INterior EnviRonments VirtuAl Synthesis

DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision

An Immersive Multi-Elevation Multi-Seasonal Dataset for 3D Reconstruction and Visualization

MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis

MegaScenes: Scene-Level View Synthesis at Scale

Self-supervised novel 2D view synthesis of large-scale scenes with efficient multi-scale voxel carving

ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

Pano2Room: Novel View Synthesis from a Single Indoor Panorama

VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation

Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Views

BelHouse3D: A Benchmark Dataset for Assessing Occlusion Robustness in 3D Point Cloud Semantic Segmentation

VEnvision3D: A Synthetic Perception Dataset for 3D Multi-Task Model Research

Learning to Reconstruct and Understand Indoor Scenes from Sparse Views

Extrapolated Urban View Synthesis Benchmark