Abstract:Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation of diffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at <a class="link-external link-https" href="https://zouzx.github.io/TriplaneGaussian/" rel="external noopener nofollow">this https URL</a>.

K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

HexPlane: A Fast Representation for Dynamic Scenes

TK-Planes: Tiered K-Planes with High Dimensional Feature Vectors for Dynamic UAV-based Scenes

DaRePlane: Direction-aware Representations for Dynamic Scene Reconstruction

R4D-planes: Remapping Planes for Novel View Synthesis and Self-Supervised Decoupling of Monocular Videos

WavePlanes: Compact Hex Planes for Dynamic Novel View Synthesis

Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

Lightplane: Highly-Scalable Components for Neural 3D Fields

PlanarRecon: Realtime 3D Plane Detection and Reconstruction from Posed Monocular Videos

PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields

Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

PlaneFusion: Real-Time Indoor Scene Reconstruction With Planar Prior

Tri$^{2}$-plane: Thinking Head Avatar via Feature Pyramid

Strivec: Sparse Tri-Vector Radiance Fields

Disentangled Generation and Aggregation for Robust Radiance Fields

Geometric Algebra Planes: Convex Implicit Neural Volumes

FastPlane: A Fully Convolutional Network for Real-time 3D Plane Segmentation

Freeplane: Unlocking Free Lunch in Triplane-Based Sparse-View Reconstruction Models

3D Gaussian Splatting for Real-Time Radiance Field Rendering

PlaneFormers: From Sparse View Planes to 3D Reconstruction

Real-Time Radiance Fields for Single-Image Portrait View Synthesis