Abstract:As a widely explored multi-modal task, 3D object grounding endeavors to localize a unique pre-existing object within a single 3D scene given a natural language description. However, such a strict setting is unnatural as it is not always possible to know whether a target object exists in a specific 3D scene. In real-world scenarios, a collection of 3D scenes is generally available, some of which may not contain the described object while some potentially contain multiple target objects. To this end, we introduce a more realistic setting, named Group-wise 3D Object Grounding, to simultaneously process a group of related 3D scenes, allowing a flexible number of target objects to exist in each scene. Instead of localizing target objects in each scene individually, we argue that ignoring the rich visual information contained in other related 3D scenes within the same group may lead to sub-optimal results. To achieve more accurate localization, we propose a baseline method named GNL3D, a Grouped Neural Listener for 3D grounding in the group-wise setting, which extends the traditional 3D object grounding pipeline with a novel language-guided consensus aggregation and distribution mechanism to explicitly exploit the intra-group visual connections. Specifically, based on context-aware spatial-semantic alignment, a language-guided consensus aggregation module is developed to aggregate the visual features of target objects in each 3D scene to form a visual consensus representation, which is then distributed and injected into a consensus-modulated feature refinement module for refining visual features, thus benefiting the subsequent multi-modal reasoning. To validate the effectiveness of the proposed method, we reorganize and enhance the ReferIt3D dataset and propose evaluation metrics to benchmark prior work and GNL3D. Extensive experiments demonstrate that GNL3D achieves state-of-the-art results on the group-wise setting and the traditional 3D object grounding task.

Dense Object Grounding in 3D Scenes

Advancing 3D Object Grounding Beyond a Single 3D Scene

Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud

Language Conditioned Spatial Relation Reasoning for 3D Object Grounding

Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding

3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds

Multi3DRefer: Grounding Text Description to Multiple 3D Objects

Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding

Rethinking 3D Dense Caption and Visual Grounding in A Unified Framework through Prompt-based Localization

DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization.

Exploiting Contextual Objects and Relations for 3D Visual Grounding.

Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention

EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding

Task-oriented Sequential Grounding in 3D Scenes

Joint Top-Down and Bottom-Up Frameworks for 3D Visual Grounding

SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

Data-Efficient 3D Visual Grounding via Order-Aware Referring

LidaRefer: Outdoor 3D Visual Grounding for Autonomous Driving with Transformers

Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph

End-to-end Visual Grounding Based on Query Text Guidance and Multi-stage Reasoning