Abstract:We propose a novel explicit dense 3D reconstruction approach that processes a set of images of a scene with sensor poses and calibrations and estimates a photo-real digital model. One of the key innovations is that the underlying volumetric representation is completely explicit in contrast to neural network-based (implicit) alternatives. We encode scenes explicitly using clear and understandable mappings of optimization variables to scene geometry and their outgoing surface radiance. We represent them using hierarchical volumetric fields stored in a sparse voxel octree. Robustly reconstructing such a volumetric scene model with millions of unknown variables from registered scene images only is a highly non-convex and complex optimization problem. To this end, we employ stochastic gradient descent (Adam) which is steered by an inverse differentiable renderer. We demonstrate that our method can reconstruct models of high quality that are comparable to state-of-the-art implicit methods. Importantly, we do not use a sequential reconstruction pipeline where individual steps suffer from incomplete or unreliable information from previous stages, but start our optimizations from uniformed initial solutions with scene geometry and radiance that is far off from the ground truth. We show that our method is general and practical. It does not require a highly controlled lab setup for capturing, but allows for reconstructing scenes with a vast variety of objects, including challenging ones, such as outdoor plants or furry toys. Finally, our reconstructed scene models are versatile thanks to their explicit design. They can be edited interactively which is computationally too costly for implicit alternatives.

Explicit 3D Reconstruction from Images with Dynamic Graph Learning and Rendering-Guided Diffusion

Multiview Textured Mesh Recovery by Differentiable Rendering

Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

Pragmatist: Multiview Conditional Diffusion Models for High-Fidelity 3D Reconstruction from Unposed Sparse Views

GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction

MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction

Self-Supervised Implicit 3D Reconstruction via RGB-D Scans.

MicroDiffusion: Implicit Representation-Guided Diffusion for 3D Reconstruction from Limited 2D Microscopy Projections

RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

Efficient Implicit Neural Reconstruction Using LiDAR

Explicit3D: Graph Network with Spatial Inference for Single Image 3D Object Detection

IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images

G3R: Gradient Guided Generalizable Reconstruction

IT3D: Improved Text-to-3D Generation with Explicit View Synthesis

GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction

ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model

ERF: Explicit Radiance Field Reconstruction From Scratch