Abstract:With the recent rise of Large Language Models (LLMs), Vision-Language Models (VLMs), and other general foundation models, there is growing potential for multimodal, multi-task embodied agents that can operate in diverse environments given only natural language as input. One such application area is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the spatial reasoning and semantic understanding required, particularly in arbitrary scenes that may contain many objects belonging to fine-grained classes. To address this challenge, we curate the largest real-world dataset for Vision and Language-guided Action in 3D Scenes (VLA-3D), consisting of over 11.5K scanned 3D indoor rooms from existing datasets, 23.5M heuristically generated semantic relations between objects, and 9.7M synthetically generated referential statements. Our dataset consists of processed 3D point clouds, semantic object and room annotations, scene graphs, navigable free space annotations, and referential language statements that specifically focus on view-independent spatial relations for disambiguating objects. The goal of these features is to aid the downstream task of navigation, especially on real-world systems where some level of robustness must be guaranteed in an open world of changing scenes and imperfect language. We benchmark our dataset with current state-of-the-art models to obtain a performance baseline. All code to generate and visualize the dataset is publicly released, see <a class="link-external link-https" href="https://github.com/HaochenZ11/VLA-3D" rel="external noopener nofollow">this https URL</a>. With the release of this dataset, we hope to provide a resource for progress in semantic 3D scene understanding that is robust to changes and one which will aid the development of interactive indoor navigation systems.

The ACM Multimedia 2024 Viual Spatial Description Grand Challenge

Generating Visual Spatial Description via Holistic 3D Scene Understanding

Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation

SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities

VidVRD 2021

VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation

LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs

V3Det Challenge 2024 on Vast Vocabulary and Open Vocabulary Object Detection: Methods and Results

V2VSSC: A 3D Semantic Scene Completion Benchmark for Perception with Vehicle to Vehicle Communication

Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding

The Robotic Vision Scene Understanding Challenge

3D Spatial Understanding in MLLMs: Disambiguation and Evaluation

SpatialBot: Precise Spatial Understanding with Vision Language Models

MapVision: CVPR 2024 Autonomous Grand Challenge Mapless Driving Tech Report

SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

PVUW 2024 Challenge on Complex Video Understanding: Methods and Results

DepthSSC: Depth-Spatial Alignment and Dynamic Voxel Resolution for Monocular 3D Semantic Scene Completion

Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Image

A Novel Image Descriptor with Aggregated Semantic Skeleton Representation for Long-term Visual Place Recognition