Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Learning by Restoring Broken 3D Geometry

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Shape Self-Correction for Unsupervised Point Cloud Understanding

Learning to Restore 3D Face from In-the-Wild Degraded Images

Scribble-Based 3D Shape Segmentation via Weakly-Supervised Learning

Self-supervised 3D Human Mesh Recovery from Noisy Point Clouds

Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering

A Machine Learning Approach to Recovery of Scene Geometry from Images

Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding

Weakly-Supervised Learning of Category-Specific 3D Object Shapes.

Self-supervised Learning of Monocular 3D Geometry Understanding with Two- and Three-View Geometric Constraints

Unsupervised 3D Shape Reconstruction by Part Retrieval and Assembly

Geometry-guided Feature Learning and Fusion for Indoor Scene Reconstruction

Self-supervised Learning of Occlusion Aware Flow Guided 3D Geometry Perception with Adaptive Cross Weighted Loss from Monocular Videos

PointGLR: Unsupervised Structural Representation Learning of 3D Point Clouds

Self-Supervised 3D Representation Learning of Dressed Humans From Social Media Videos

Exploring Geometry-aware Contrast and Clustering Harmonization for Self-supervised 3D Object Detection.

Learning to Recover 3D Scene Shape from a Single Image

Reassembling Broken Objects using Breaking Curves

Learning 3D Object Shape and Layout without 3D Supervision