Abstract:Thanks to its precise spatial referencing, 3D point cloud visual grounding is essential for deep understanding and dynamic interaction in 3D environments, encompassing 3D Referring Expression Comprehension (3DREC) and Segmentation (3DRES). We argue that 3DREC and 3DRES should be unified in one framework, which is also a natural progression in the community. To explain, 3DREC help 3DRES locate the referent, while 3DRES also facilitate 3DREC via more fine-grained language-visual alignment. To achieve this, this paper takes the initiative step to integrate 3DREC and 3DRES into a unified framework, termed 3D Referring Transformer (3DRefTR). Its key idea is to build upon a mature 3DREC model and leverage ready query embeddings and visual tokens from the 3DREC model to construct a dedicated mask branch. Specially, we propose Superpoint Mask Branch, which serves a dual purpose: i) By harnessing on the inherent association between the superpoints and point cloud, it eliminates the heavy computational overhead on the high-resolution visual features for upsampling; ii) By leveraging the heterogeneous CPU-GPU parallelism, while the GPU is occupied generating visual and language tokens, the CPU concurrently produces superpoints, equivalently accomplishing the upsampling computation. This elaborate design enables 3DRefTR to achieve both well-performing 3DRES and 3DREC capacities with only a 6% additional latency compared to the original 3DREC model. Empirical evaluations affirm the superiority of 3DRefTR. Specifically, on the ScanRefer dataset, 3DRefTR surpasses the state-of-the-art 3DRES method by 12.43% in mIoU and improves upon the SOTA 3DREC method by 0.6% Acc@0.25IoU. The codes and models will be released soon.

Torch-Points3D: A Modular Multi-Task Frameworkfor Reproducible Deep Learning on 3D Point Clouds

Pass3d: Precise And Accelerated Semantic Segmentation For 3d Point Cloud

PnP-3D: A Plug-and-Play for 3D Point Clouds

DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration

TorchSparse plus plus : Efficient Point Cloud Engine

A Unified Framework for 3D Point Cloud Visual Grounding

PointHR: Exploring High-Resolution Architectures for 3D Point Cloud Segmentation

Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

Empir3D : A Framework for Multi-Dimensional Point Cloud Assessment

PointResNet: Residual Network for 3D Point Cloud Segmentation and Classification

EasyTrack: Efficient and Compact One-stream 3D Point Clouds Tracker

Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysis

Deep Point Cloud Reconstruction

PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection

PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud

Accelerating DNN-based 3D point cloud processing for mobile computing

PointInst3D: Segmenting 3D Instances by Points

Deep Learning for 3D Point Cloud Understanding: A Survey

P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoising

Triangle-Net: Towards Robustness in Point Cloud Learning

Point Transformer V3: Simpler, Faster, Stronger