Abstract:As a widely explored multi-modal task, 3D object grounding endeavors to localize a unique pre-existing object within a single 3D scene given a natural language description. However, such a strict setting is unnatural as it is not always possible to know whether a target object exists in a specific 3D scene. In real-world scenarios, a collection of 3D scenes is generally available, some of which may not contain the described object while some potentially contain multiple target objects. To this end, we introduce a more realistic setting, named Group-wise 3D Object Grounding, to simultaneously process a group of related 3D scenes, allowing a flexible number of target objects to exist in each scene. Instead of localizing target objects in each scene individually, we argue that ignoring the rich visual information contained in other related 3D scenes within the same group may lead to sub-optimal results. To achieve more accurate localization, we propose a baseline method named GNL3D, a Grouped Neural Listener for 3D grounding in the group-wise setting, which extends the traditional 3D object grounding pipeline with a novel language-guided consensus aggregation and distribution mechanism to explicitly exploit the intra-group visual connections. Specifically, based on context-aware spatial-semantic alignment, a language-guided consensus aggregation module is developed to aggregate the visual features of target objects in each 3D scene to form a visual consensus representation, which is then distributed and injected into a consensus-modulated feature refinement module for refining visual features, thus benefiting the subsequent multi-modal reasoning. To validate the effectiveness of the proposed method, we reorganize and enhance the ReferIt3D dataset and propose evaluation metrics to benchmark prior work and GNL3D. Extensive experiments demonstrate that GNL3D achieves state-of-the-art results on the group-wise setting and the traditional 3D object grounding task.

Advancing 3D Object Grounding Beyond a Single 3D Scene

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Task-oriented Sequential Grounding in 3D Scenes

Dense Object Grounding in 3D Scenes

Multi3DRefer: Grounding Text Description to Multiple 3D Objects

Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding

SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

Grounded 3D-LLM with Referent Tokens

Exploiting Contextual Objects and Relations for 3D Visual Grounding.

LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent

Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention

Language Conditioned Spatial Relation Reasoning for 3D Object Grounding

Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding

G$^3$-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding

Revisiting 3D Visual Grounding with Context-aware Feature Aggregation

Multi-Task Domain Adaptation for Language Grounding with 3D Objects

3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding

Cross-Modal Match for Language Conditioned 3D Object Grounding

Data-Efficient 3D Visual Grounding via Order-Aware Referring

Mono3DVG: 3D Visual Grounding in Monocular Images

Joint Top-Down and Bottom-Up Frameworks for 3D Visual Grounding