Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Category-Sensitive Incremental Learning for Image-Based 3D Shape Reconstruction

Single-view 3D Mesh Reconstruction for Seen and Unseen Categories

A Knowledge-Based Strategy for Object Recognition and Reconstruction

Deep 3D Shape Reconstruction from Single-View Sketch Image

When 3D Reconstruction Meets Ubiquitous RGB-D Images

CIGNet: Category-and-Intrinsic-Geometry Guided Network for 3D Coarse-to-fine Reconstruction.

Reconstructing 3D Model from Single-View Sketch with Deep Neural Network.

3D Shape Knowledge Graph for Cross‐domain 3D Shape Retrieval

Single View 3D Reconstruction with Category Information Learning

CVRecon: Rethinking 3D Geometric Feature Learning for Neural Reconstruction.

3D Surface Reconstruction in the Wild by Deforming Shape Priors from Synthetic Data

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks

Three-dimensional reconstruction of an object from a single image using deep convolutional neural networks

Learning single-image 3D reconstruction by generative modelling of shape, pose and shading

Zero-Shot Single-View Point Cloud Reconstruction Via Cross-Category Knowledge Transferring

DV-Net: Dual-view Network for 3D Reconstruction by Fusing Multiple Sets of Gated Control Point Clouds

Deep Optimized Priors for 3D Shape Modeling and Reconstruction

Learning Shape Priors for Single-View 3D Completion and Reconstruction

Learning Free-Form Deformations for 3D Object Reconstruction