Abstract:We propose a novel 3d colored shape reconstruction method from a single RGB image through diffusion model. Diffusion models have shown great development potentials for high-quality 3D shape generation. However, most existing work based on diffusion models only focus on geometric shape generation, they cannot either accomplish 3D reconstruction from a single image, or produce 3D geometric shape with color information. In this work, we propose to reconstruct a 3D colored shape from a single RGB image through a novel conditional diffusion model. The reverse process of the proposed diffusion model is consisted of three modules, shape prediction module, color prediction module and NeRF-like rendering module. In shape prediction module, the reference RGB image is first encoded into a high-level shape feature and then the shape feature is utilized as a condition to predict the reverse geometric noise in diffusion model. Then the color of each 3D point updated in shape prediction module is predicted by color prediction module. Finally, a NeRF-like rendering module is designed to render the colored point cloud predicted by the former two modules to 2D image space to guide the training conditioned only on a reference image. As far as the authors know, the proposed method is the first diffusion model for 3D colored shape reconstruction from a single RGB image. Experimental results demonstrate that the proposed method achieves competitive performance on colored 3D shape reconstruction, and the ablation study validates the positive role of the color prediction module in improving the reconstruction quality of 3D geometric point cloud.

Shape Reconstruction Within an Optical Diffusion Model.

Shape from defocus via diffusion

Optical Image Reconstruction of Non-Scattering and Low Scattering Heterogeneities in Turbid Media Based on the Diffusion Approximation Model

Shape Reconstruction Based on a New Blurring Model at the Micro/Nanometer Scale

Curved Diffusion: A Generative Model With Optical Geometry Control

Depth reconstruction with Fresnel diffraction

3D Colored Shape Reconstruction from a Single RGB Image through Diffusion

Nanoscale Depth Reconstruction from Defocus: Within an Optical Diffraction Model.

A higher order diffusion model for three-dimensional photon migration and image reconstruction in optical tomography.

A Reconstruction Method of Electromagnetic Scattering Target Based on Diffusion Model

HoloDiffusion: Sparse Digital Holographic Reconstruction via Diffusion Modeling

Computational imaging without a computer: seeing through random diffusers at the speed of light

Modeling and Rendering of Heterogeneous Translucent Materials Using the Diffusion Equation

MicroDiffusion: Implicit Representation-Guided Diffusion for 3D Reconstruction from Limited 2D Microscopy Projections

Three-dimensional Modelling of Blur Property for Conventional Optical Microscopes

Computational Reconstruction Algorithm for Integral Imaging Based on Diffraction Tracing

Nano Scale Depth Reconstruction Based on Optical Diffraction

A reconstruction method for diffuse optical tomography by grouping of source-detector configuration

Fast Depth Reconstruction With A Defocus Model On Micro Scale

TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation