Abstract:By formulating the data generation as a sequence procedure of denoising autoencoding, diffusion models have achieved superior in-painting performance on image data and beyond. Nevertheless, it is not trivial when capitalizing on diffusion models to generate missing 3D points. The difficulty originates from the intrinsic structure where 3D point cloud is a set of unordered and irregular coordinates. That motivates us to delve into the 3D structural information for designing point cloud encoder-decoder and shape latent generator, to precisely formulate the latent distribution of the complete point cloud and partial observation. In this paper, we propose Point cloud completion with Latent Diffusion Models (PointLDM), a new approach that leverages the conditional denoising diffusion probabilistic modeling (DDPM) in the 3D latent space for shape reconstruction. The architecture of PointLDM consists of a transformer-based variational auto-encoder (VAE) to model the complete shape latent, and a diffusion network for shape latent prediction. The encoder of VAE exploits both of global shape latent and local point features in shape distribution learning. With the learnt shape latent, the decoder first decodes the shape latent into coarse points, and then recovers the fine-grained details around each coarse point by deforming a 2D grid. To reconstruct the shape latent from partial observation, the diffusion network treats the partial observation as the conditional input and generates the shape latent via DDPM. Extensive experiments conducted on MVP, Completion3D, and KITTI quantitatively and qualitatively demonstrate the efficacy of PointLDM over the state-of-the-art shape completion approaches.

POC-SLT: Partial Object Completion with SDF Latent Transformers

ShapeFormer: Transformer-based Shape Completion Via Sparse Representation

SC-Diff: 3D Shape Completion with Latent Diffusion Models

Autoregressive 3D Shape Completion via Sphere-Guided Disentangled Representation.

Implicit Shape Completion via Adversarial Shape Priors

Deformable Shape Completion with Graph Convolutional Autoencoders

SDPT: Semantic-Aware Dimension-Pooling Transformer for Image Segmentation

Learning 3D Shape Latent for Point Cloud Completion

3DPCTN: Two 3D Local-Object Point-Cloud-Completion Transformer Networks Based on Self-Attention and Multi-Resolution

PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer

Gsformer: geometric-spatial transformer on point cloud completion

Latent Feature-Aware and Local Structure-Preserving Network for 3D Completion from a Single Depth View

Point Cloud Completion Via Skeleton-Detail Transformer

Leveraging Shape Completion for 3D Siamese Tracking

AShapeFormer: Semantics-Guided Object-Level Active Shape Encoding for 3D Object Detection Via Transformers

3D-TRANS: 3D Hierarchical Transformer for Shape Correspondence Learning.

3D Shape Completion on Unseen Categories: A Weakly-Supervised Approach

Structure-Aware Cross-Modal Transformer for Depth Completion

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

Improving Object Grasp Performance Via Transformer-Based Sparse Shape Completion

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion