Abstract:We propose a novel 3d colored shape reconstruction method from a single RGB image through diffusion model. Diffusion models have shown great development potentials for high-quality 3D shape generation. However, most existing work based on diffusion models only focus on geometric shape generation, they cannot either accomplish 3D reconstruction from a single image, or produce 3D geometric shape with color information. In this work, we propose to reconstruct a 3D colored shape from a single RGB image through a novel conditional diffusion model. The reverse process of the proposed diffusion model is consisted of three modules, shape prediction module, color prediction module and NeRF-like rendering module. In shape prediction module, the reference RGB image is first encoded into a high-level shape feature and then the shape feature is utilized as a condition to predict the reverse geometric noise in diffusion model. Then the color of each 3D point updated in shape prediction module is predicted by color prediction module. Finally, a NeRF-like rendering module is designed to render the colored point cloud predicted by the former two modules to 2D image space to guide the training conditioned only on a reference image. As far as the authors know, the proposed method is the first diffusion model for 3D colored shape reconstruction from a single RGB image. Experimental results demonstrate that the proposed method achieves competitive performance on colored 3D shape reconstruction, and the ablation study validates the positive role of the color prediction module in improving the reconstruction quality of 3D geometric point cloud.

A Weighted Color MRF Model for 3D Reconstruction from a Single Image

3d Reconstruction With Multi-View Texture Mapping

Single View 3D Reconstruction Based on Improved RGB-D Image

CRF-Based Reconstruction from Narrow-Baseline Image Sequences.

Colorful 3D Reconstruction at High Resolution Using Multi-View Representation

Robust 3D Reconstruction with an RGB-D Camera

A 3d Reconstruction System For Large Scene Based On Rgb-D Image

3D Reconstruction of Dense Model Based on the Sparse Frames Using RGBD Camera

3D colored object reconstruction from a single view image through diffusion

3D Colored Shape Reconstruction from a Single RGB Image through Diffusion

Visual Odometry Based 3D-Reconstruction

An improved MRF model for robust color guided depth up-sampling

An Active Volumetric Model for 3D Reconstruction

Semantic Reconstruction based on RGB Image and Sparse Depth

3D Human Reconstruction from A Single Depth Image

Sparse RGB-D Images Create a Real Thing: A Flexible Voxel Based 3D Reconstruction Pipeline for Single Object

Dense 3D Reconstruction Combining Depth and RGB Information

Deep-learning Based Single-Shot 3D Reconstruction with Simulated Color-Crosstalk and Randomized Extrinsics

3D Reconstruction from a Single Still Image Based on Monocular Vision of an Uncalibrated Camera

Im2Avatar: Colorful 3D Reconstruction from a Single Image

A revisit to MRF-based depth map super-resolution and enhancement