Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Learning Category-Specific 3d Shape Models From Weakly Labeled 2d Images

Weakly-Supervised Learning of Category-Specific 3D Object Shapes.

3D Shape Completion on Unseen Categories:A Weakly-supervised Approach

3D Shape Completion on Unseen Categories: A Weakly-Supervised Approach

Scribble-Based 3D Shape Segmentation via Weakly-Supervised Learning

Category Modeling from Just a Single Labeling: Use Depth Information to Guide the Learning of 2D Models

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Weakly Supervised 3D Instance Segmentation without Instance-level Annotations

A Unified Feature Representation and Learning Framework for 3D Shape

Weakly Supervised Classification Model for Zero‐shot Semantic Segmentation

Category-Sensitive Incremental Learning for Image-Based 3D Shape Reconstruction

Weaklier Supervised Semantic Segmentation with Only One Image Level Annotation Per Category.

Towards A Weakly Supervised Framework for 3D Point Cloud Object Detection and Annotation

Semi-supervised 3D shape segmentation with multilevel consistency and part substitution

Weakly Supervised Fine-Grained Categorization With Part-Based Image Representation

PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation

Learning 3D Shape Completion under Weak Supervision

Visual Boundary-Guided Pseudo-Labeling for Weakly Supervised 3D Point Cloud Segmentation in Indoor Environments

Learning to group and label fine-grained shape components

Shape2Scene: 3D Scene Representation Learning Through Pre-training on Shape Data

Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels