Abstract:Point cloud videos effectively capture real-world spatial geometries and temporal dynamics, which are essential for enabling intelligent agents to understand the dynamically changing 3D world we live in. Although static 3D point cloud processing has witnessed significant advancements, designing an effective 4D point cloud video backbone remains challenging, mainly due to the irregular and unordered distribution of points and temporal inconsistencies across frames. Moreover, recent state-of-the-art 4D backbones predominantly rely on transformer-based architectures, which commonly suffer from large computational costs due to their quadratic complexity, particularly when processing long video sequences. To address these challenges, we propose a novel 4D point cloud video understanding backbone based on the recently advanced State Space Models (SSMs). Specifically, our backbone begins by disentangling space and time in raw 4D sequences, and then establishing spatio-temporal correlations using our newly developed Intra-frame Spatial Mamba and Inter-frame Temporal Mamba blocks. The Intra-frame Spatial Mamba module is designed to encode locally similar or related geometric structures within a certain temporal searching stride, which can effectively capture short-term dynamics. Subsequently, these locally correlated tokens are delivered to the Inter-frame Temporal Mamba module, which globally integrates point features across the entire video with linear complexity, further establishing long-range motion dependencies. Experimental results on human action recognition and 4D semantic segmentation tasks demonstrate the superiority of our proposed method. Especially, for long video sequences, our proposed Mamba-based method has an 87.5% GPU memory reduction, 5.36 times speed-up, and much higher accuracy (up to +10.4%) compared with transformer-based counterparts on MSR-Action3D dataset.

LeaF: Learning Frames for 4D Point Cloud Sequence Understanding

Local Deep Feature Learning Framework for 3D Shape.

Learning Temporal Variations for 4D Point Cloud Segmentation

FeatFlow: Learning Geometric Features for 3D Motion Estimation

Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds

Local Optimized and Scalable Frame-to-model SLAM

PA-Pose: Partial Point Cloud Fusion Based on Reliable Alignment for 6D Pose Tracking

Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning

Scalable Learning Framework for Traversable Region Detection Fusing with Appearance and Geometrical Information

Equivariant Local Reference Frames for Unsupervised Non-rigid Point Cloud Shape Correspondence

Learning Spatial and Temporal Variations for 4D Point Cloud Segmentation

X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos Through Cross-modal Knowledge Transfer

MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models

3Dmotion-Net: Learning Continuous Flow Function for 3D Motion Prediction.

Beyond Skeletons: Integrative Latent Mapping for Coherent 4D Sequence Generation

A 4D strong spatio-temporal feature learning network for behavior recognition of point cloud sequences

PointGLR: Unsupervised Structural Representation Learning of 3D Point Clouds

Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning

NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understanding

Learning to Estimate 3-D States of Deformable Linear Objects from Single-Frame Occluded Point Clouds

S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points