Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Colorful 3D Reconstruction at High Resolution Using Multi-View Representation

High-fidelity 3D Face Reconstruction with Multi-Scale Details

Im2Avatar: Colorful 3D Reconstruction from a Single Image

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture

3D colored object reconstruction from a single view image through diffusion

3D Colored Shape Reconstruction from a Single RGB Image through Diffusion

Multi-View Depth Map Sampling for 3D Reconstruction of Natural Scene

A 3D Reconstruction Method with Color Reproduction from Multi-band and Multi-view Images

UNeR3D: Versatile and Scalable 3D RGB Point Cloud Generation from 2D Images in Unsupervised Reconstruction

High-resolution 3D reconstruction for complex color scenes with structured light

Learning To Reconstruct High-Quality 3d Shapes With Cascaded Fully Convolutional Networks

Visual Odometry Based 3D-Reconstruction

High-Resolution Volumetric Reconstruction for Clothed Humans

Color-NeuS: Reconstructing Neural Implicit Surfaces with Color

Semi-supervised Single-view 3D Reconstruction via Multi Shape Prior Fusion Strategy and Self-Attention

High-Quality RGB-D Reconstruction via Multi-View Uncalibrated Photometric Stereo and Gradient-SDF

A Coarse-to-Fine Transformer-Based Network for 3D Reconstruction from Non-Overlapping Multi-View Images

Real-Time Globally Consistent Dense 3D Reconstruction with Online Texturing.