Abstract:Thanks to its precise spatial referencing, 3D point cloud visual grounding is essential for deep understanding and dynamic interaction in 3D environments, encompassing 3D Referring Expression Comprehension (3DREC) and Segmentation (3DRES). We argue that 3DREC and 3DRES should be unified in one framework, which is also a natural progression in the community. To explain, 3DREC help 3DRES locate the referent, while 3DRES also facilitate 3DREC via more fine-grained language-visual alignment. To achieve this, this paper takes the initiative step to integrate 3DREC and 3DRES into a unified framework, termed 3D Referring Transformer (3DRefTR). Its key idea is to build upon a mature 3DREC model and leverage ready query embeddings and visual tokens from the 3DREC model to construct a dedicated mask branch. Specially, we propose Superpoint Mask Branch, which serves a dual purpose: i) By harnessing on the inherent association between the superpoints and point cloud, it eliminates the heavy computational overhead on the high-resolution visual features for upsampling; ii) By leveraging the heterogeneous CPU-GPU parallelism, while the GPU is occupied generating visual and language tokens, the CPU concurrently produces superpoints, equivalently accomplishing the upsampling computation. This elaborate design enables 3DRefTR to achieve both well-performing 3DRES and 3DREC capacities with only a 6% additional latency compared to the original 3DREC model. Empirical evaluations affirm the superiority of 3DRefTR. Specifically, on the ScanRefer dataset, 3DRefTR surpasses the state-of-the-art 3DRES method by 12.43% in mIoU and improves upon the SOTA 3DREC method by 0.6% Acc@0.25IoU. The codes and models will be released soon.

TransUPR: A Transformer-based Uncertain Point Refiner for LiDAR Point Cloud Semantic Segmentation

TransUPR: A Transformer-based Plug-and-Play Uncertain Point Refiner For LiDAR Point Cloud Semantic Segmentation

Refined Division Features Based on Transformer for Semantic Image Segmentation

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

PU-CTG: A Point Cloud Upsampling Network Using Transformer Fusion and GRU Correction

Position-Guided Point Cloud Panoptic Segmentation Transformer

A Transformer-based Real-time LiDAR Semantic Segmentation Method for Restricted Mobile Devices

Stratified Transformer for 3D Point Cloud Segmentation

PU-Transformer: Point Cloud Upsampling Transformer

Point cloud upsampling via a coarse-to-fine network with transformer-encoder

TULIP: Transformer for Upsampling of LiDAR Point Clouds

Point Cloud Upsampling via Cascaded Refinement Network

High-Performance Feature Extraction Network for Point Cloud Semantic Segmentation

A Unified Framework for 3D Point Cloud Visual Grounding

DOCTR: Disentangled Object-Centric Transformer for Point Scene Understanding

SemAffiNet: Semantic-Affine Transformation for Point Cloud Segmentation

STransFuse: Fusing Swin Transformer and Convolutional Neural Network for Remote Sensing Image Semantic Segmentation

PointRas: Uncertainty-Aware Multi-Resolution Learning for Point Cloud Segmentation

Query Refinement Transformer for 3D Instance Segmentation

Dynamic clustering transformer network for point cloud segmentation

D2T-Net: A dual-domain transformer network exploiting spatial and channel dimensions for semantic segmentation of urban mobile laser scanning point clouds