Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Image Augment Self-Supervised Network for Hand-Drawn Sketch 3D Reconstruction

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Sketch-R2CNN: an Attentive Network for Vector Sketch Recognition

Geometric Object 3D Reconstruction from Single Line Drawing Image Based on a Network for Classification and Sketch Extraction

Model-based 3D Hand Reconstruction via Self-Supervised Learning

Deep Self-Supervised Representation Learning for Free-Hand Sketch

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Learning 3D Face Reconstruction from a Single Sketch

Self-supervised single-image 3D face reconstruction method based on attention mechanism and attribute refinement

Sketch-Specific Data Augmentation for Freehand Sketch Recognition

Unsupervised 3D Reconstruction from a Single Image via Adversarial Learning

Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches

Three cheers for the pain pump?

Video Supervised for 3D Reconstruction from Single Image

Self-Supervised 3D Mesh Reconstruction from Single Images

Self-supervised Spatial Reasoning on Multi-View Line Drawings

Deep3DSketch+: Obtaining Customized 3D Model by Single Free-Hand Sketch through Deep Learning

Deep3DSketch-im: rapid high-fidelity AI 3D model generation by single freehand sketches

Self-Supervised Sketch-to-Image Synthesis

Enhanced Multi-Scale Attention-Driven 3D Human Reconstruction from Single Image

Self-supervised reflectance-guided 3d shape reconstruction from single-view images