Abstract:Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation of diffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at <a class="link-external link-https" href="https://zouzx.github.io/TriplaneGaussian/" rel="external noopener nofollow">this https URL</a>.

AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos

A Submap Joining Algorithm for 3D Reconstruction Using an RGB-D Camera Based on Point and Plane Features

In-Hand 3D Object Reconstruction from a Monocular RGB Video

MonoPlane: Exploiting Monocular Geometric Cues for Generalizable 3D Plane Reconstruction

PlanarRecon: Realtime 3D Plane Detection and Reconstruction from Posed Monocular Videos

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone

Mobile3DScanner: an Online 3D Scanner for High-quality Object Reconstruction with a Mobile Device

Real-time dense 3D reconstruction and camera tracking via embedded planes representation

UniPlane: Unified Plane Detection and Reconstruction from Posed Monocular Videos

Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

Tri$^{2}$-plane: Thinking Head Avatar via Feature Pyramid

PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo

Real-time Dense Reconstruction of Tissue Surface from Stereo Optical Video

SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos

Planar Reconstruction of Indoor Scenes from Sparse Views and Relative Camera Poses

Real-Time Trust Region Ground Plane Segmentation For Monocular Mobile Robots

R4D-planes: Remapping Planes for Novel View Synthesis and Self-Supervised Decoupling of Monocular Videos

TranSplat: Generalizable 3D Gaussian Splatting from Sparse Multi-View Images with Transformers

TriSpaSurf: A Triple-View Outline Guided 3D Surface Reconstruction of Vehicles from Sparse Point Cloud

Three-dimensional point cloud plane segmentation in both structured and unstructured environments

PlaneFusion: Real-Time Indoor Scene Reconstruction With Planar Prior