Abstract:Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation of diffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at <a class="link-external link-https" href="https://zouzx.github.io/TriplaneGaussian/" rel="external noopener nofollow">this https URL</a>.

Blaze3DM: Marry Triplane Representation with Diffusion for 3D Medical Inverse Problem Solving

Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models

3D MedDiffusion: A 3D Medical Diffusion Model for Controllable and High-quality Medical Image Generation

DiffusionBlend: Learning 3D Image Prior through Position-aware Diffusion Score Blending for 3D Computed Tomography Reconstruction

Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis

Denoising diffusion probabilistic models for 3D medical image generation

DMCVR: Morphology-Guided Diffusion Model for 3D Cardiac Volume Reconstruction

2.5D Multi-view Averaging Diffusion Model for 3D Medical Image Translation: Application to Low-count PET Reconstruction with CT-less Attenuation Correction

3D Neural Field Generation using Triplane Diffusion

3D MRI Synthesis with Slice-Based Latent Diffusion Models: Improving Tumor Segmentation Tasks in Data-Scarce Regimes

Generative Enhancement for 3D Medical Images

DIFR3CT: Latent Diffusion for Probabilistic 3D CT Reconstruction from Few Planar X-Rays

Slice-Consistent 3D Volumetric Brain CT-to-MRI Translation with 2D Brownian Bridge Diffusion Model

Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis

Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

MediSyn: Text-Guided Diffusion Models for Broad Medical 2D and 3D Image Synthesis

EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior

PrideDiff: Physics-Regularized Generalized Diffusion Model for CT Reconstruction

Wonder3D: Single Image to 3D Using Cross-Domain Diffusion

3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation

Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views