Abstract:3D referring segmentation is an emerging and challenging vision-language task that aims to segment the object described by a natural language expression in a point cloud scene. The key challenge behind this task is vision-language feature fusion and alignment. In this work, we propose RefMask3D to explore the comprehensive multi-modal feature interaction and understanding. First, we propose a Geometry-Enhanced Group-Word Attention to integrate language with geometrically coherent sub-clouds through cross-modal group-word attention, which effectively addresses the challenges posed by the sparse and irregular nature of point clouds. Then, we introduce a Linguistic Primitives Construction to produce semantic primitives representing distinct semantic attributes, which greatly enhance the vision-language understanding at the decoding stage. Furthermore, we introduce an Object Cluster Module that analyzes the interrelationships among linguistic primitives to consolidate their insights and pinpoint common characteristics, helping to capture holistic information and enhance the precision of target identification. The proposed RefMask3D achieves new state-of-the-art performance on 3D referring segmentation, 3D visual grounding, and also 2D referring image segmentation. Especially, RefMask3D outperforms previous state-of-the-art method by a large margin of 3.16% mIoU} on the challenging ScanRefer dataset. Code is available at <a class="link-external link-https" href="https://github.com/heshuting555/RefMask3D" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper primarily addresses the key challenge in the 3D referring segmentation task—fusion and alignment of visual and linguistic features. Specifically, the paper proposes a new model named RefMask3D, which aims to improve existing methods through the following approaches: 1. **Geometry-Enhanced Group-Word Attention**: - A geometry-enhanced group-word attention mechanism is proposed, which performs cross-modal attention at each stage of point cloud encoding, integrating linguistic features with local sub-point clouds (i.e., groups) that have geometric adjacency. - This approach effectively addresses the challenges posed by the sparse and irregular distribution of point clouds, reduces noise introduced by direct point-word association, and leverages the inherent geometric relationships within the point cloud. 2. **Linguistic Primitives Construction**: - A strategy called Linguistic Primitives Construction (LPC) is designed to generate primitives representing different semantic attributes (such as shape, color, size, etc.). - These primitives can acquire corresponding attributes through interaction with specific linguistic information. Inputting these semantically rich primitives into the decoder can enhance the network's attention to diverse semantics within the point cloud, thereby significantly improving the accuracy of target object recognition. 3. **Object Cluster Module**: - An object cluster module is introduced to capture holistic information and generate object embeddings for segmenting the target object. - This module analyzes the interrelationships among linguistic primitives to unify their insights and determine common features, which helps deepen the overall understanding of linguistic and visual information. Through the above methods, RefMask3D not only achieves new state-of-the-art performance in the 3D referring segmentation task but also performs excellently in 3D visual localization and 2D referring image segmentation tasks. Particularly on the ScanRefer dataset, RefMask3D improves the mean Intersection over Union (mIoU) by 3.16% compared to the previous best method, demonstrating its strong capability in handling complex multimodal tasks.

RefMask3D: Language-Guided Transformer for 3D Referring Segmentation

A Unified Framework for 3D Point Cloud Visual Grounding

Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

X-RefSeg3D: Enhancing Referring 3D Instance Segmentation Via Structured Cross-Modal Graph Neural Networks

Mask Grounding for Referring Image Segmentation

XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

Language-Aware Spatial-Temporal Collaboration for Referring Video Segmentation

CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation

LESS: Label-Efficient and Single-Stage Referring 3D Segmentation

SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process

Position-Guided Point Cloud Panoptic Segmentation Transformer

Language-Aware Vision Transformer for Referring Segmentation

OpenMask3D: Open-Vocabulary 3D Instance Segmentation

Linguistic Query-Guided Mask Generation for Referring Image Segmentation

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

RefSAM3D: Adapting SAM with Cross-modal Reference for 3D Medical Image Segmentation

Reimagining 3D Visual Grounding: Instance Segmentation and Transformers for Fragmented Point Cloud Scenarios.