Abstract:Point clouds are a very efficient way to represent volumetric data in medical imaging. First, they do not occupy resources for empty spaces and therefore can avoid trade-offs between resolution and field-of-view for voxel-based 3D convolutional networks (CNNs) - leading to smaller and robust models. Second, they provide a modality agnostic representation of anatomical surfaces and shapes to avoid domain gaps for generic geometric models. Third, they remove identifiable patient-specific information and may increase privacy preservation when publicly sharing data. Despite their benefits, point clouds are still underexplored in medical imaging compared to volumetric 3D CNNs and vision transformers. To date both datasets and stringent studies on comparative strengths and weaknesses of methodological choices are missing. Interactions and information exchange of spatially close points - e.g. through k-nearest neighbour graphs in edge convolutions or point transformations - within points clouds are crucial for learning geometrically meaningful features but may incur computational bottlenecks. This work presents a hybrid approach that combines point-wise operations with intermediate differentiable rasterisation and dense localised CNNs. For deformable point cloud registration, we devise an early fusion scheme for coordinate features that joins both clouds within a common reference frame and is coupled with an inverse consistent, two-step alignment architecture. Our extensive experiments on three different datasets for segmentation and registration demonstrate that our method, PointVoxelFormer, enables very compact models that excel with threefold speed-ups, fivefold memory reduction and over 30% registration error reduction against edge convolutions and other state-of-the-art models in geometric deep learning.

MPVNN: Multi-resolution Point-Voxel Non-parametric Network for 3D Point Cloud Processing

Continuous Volumetric Convolution Network with Self-Learning Kernels for Point Clouds

Multi Point-Voxel Convolution (MPVConv) for Deep Learning on Point Clouds

Multi-View PointNet for 3D Scene Understanding

VV-Net: Voxel VAE Net with Group Convolutions for Point Cloud Segmentation

Point-Voxel Based Geometry-Adaptive Network for 3D Point Cloud Analysis

Multi Voxel-Point Neurons Convolution (MVPConv) for Fast and Accurate 3D Deep Learning

Point-Voxel CNN for Efficient 3D Deep Learning

AVFP-MVX: Multimodal VoxelNet with Attention Mechanism and Voxel Feature Pyramid

VIC-Net: Voxelization Information Compensation Network for Point Cloud 3D Object Detection

VP-Net: Voxels As Points for 3-D Object Detection.

PointVoxelFormer -- Reviving point cloud networks for 3D medical imaging

VTPNet for 3D deep learning on point cloud

PSVMLP: Point and Shifted Voxel MLP for 3D Deep Learning

VP-Net: Voxels as Points for 3D Object Detection

Sparse Point-Voxel Aggregation Network for Efficient Point Cloud Semantic Segmentation

PointVGG: Graph Convolutional Network with Progressive Aggregating Features on Point Clouds.

PVConvNet: Pixel-Voxel Sparse Convolution for multimodal 3D object detection

RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation

PointMM: Point Cloud Semantic Segmentation CNN under Multi-Spatial Feature Encoding and Multi-Head Attention Pooling