Abstract:3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to decompose complex natural language queries into semantic constituents and employs a visual grounding tool, such as OpenScene or LERF, to identify objects in a 3D scene. The LLM then evaluates the spatial and commonsense relations among the proposed objects to make a final grounding decision. Our method does not require any labeled training data and can generalize to novel 3D scenes and arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and demonstrate state-of-the-art zero-shot grounding accuracy. Our findings indicate that LLMs significantly improve the grounding capability, especially for complex language queries, making LLM-Grounder an effective approach for 3D vision-language tasks in robotics. Videos and interactive demos can be found on the project website <a class="link-external link-https" href="https://chat-with-nerf.github.io/" rel="external noopener nofollow">this https URL</a> .

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in 3D visual localization tasks, existing CLIP - based models exhibit "bag - of - words" behavior when dealing with complex natural language queries, that is, they ignore the semantic structure in the text input, especially complex spatial relationships. For example, when given a query containing complex spatial relationships, such as "the chair between a table and a window", these models may incorrectly highlight the table and the window instead of the target object - the chair. This limits the robot's ability to navigate in the home environment, manipulate objects, and answer questions based on the environment. To overcome this challenge, the authors propose a new zero - sample, open - vocabulary 3D visual localization pipeline - LLM - Grounder. This method uses large - language models (LLMs) as agents to decompose complex natural language queries, and collects information by interacting with tools (such as object finders and landmark finders). Finally, it uses spatial and common - sense knowledge to reflect on the collected feedback to make the final localization decision. Specifically, the main contributions of LLM - Grounder are as follows: 1. **Using LLMs to improve the capabilities of zero - sample open - vocabulary methods in 3D visual localization tasks**: By using LLMs as agents, the performance of CLIP - based 3D visual localization methods (such as LERF and OpenScene) in zero - sample settings can be significantly improved. 2. **Achieving state - of - the - art zero - sample localization accuracy on the ScanRefer benchmark**: Experimental results show that LLM - Grounder achieves the highest zero - sample localization accuracy on the ScanRefer benchmark without using any labeled data. 3. **Discovering that LLMs are more effective in handling more complex language queries**: As the complexity of language queries increases, the role of LLMs in improving localization capabilities becomes more obvious. Through these contributions, LLM - Grounder provides an effective solution for 3D vision - language tasks, especially suitable for robotic applications that need to understand complex environments and respond to dynamic queries.

LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent

VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

Grounded 3D-LLM with Referent Tokens

LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding

SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding

Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learning

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding

Grounding Language with Visual Affordances over Unstructured Data

GeoGround: A Unified Large Vision-Language Model. for Remote Sensing Visual Grounding

Learning Visual Grounding from Generative Vision and Language Model

GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions

LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models

Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination

Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision

GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection

Towards CLIP-driven Language-free 3D Visual Grounding Via 2D-3D Relational Enhancement and Consistency

ViewInfer3D: 3D Visual Grounding Based on Embodied Viewpoint Inference

Advancing 3D Object Grounding Beyond a Single 3D Scene

G$^3$-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding