Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

High-level semantic feature for 3D shape based on deep belief networks

Learning High-Level Feature by Deep Belief Networks for 3-D Model Retrieval and Recognition.

Local Deep Feature Learning Framework for 3D Shape.

A Unified Feature Representation and Learning Framework for 3D Shape

Deep Learning Representation using Autoencoder for 3D Shape Retrieval

DeepShape: Deep-Learned Shape Descriptor for 3D Shape Retrieval

A Dimensional Reduction Guiding Deep Learning Architecture for 3D Shape Retrieval

Learning Discriminative and Generative Shape Embeddings for Three-Dimensional Shape Retrieval

3D Shape Retrieval Based on Laplace Operator and Joint Bayesian Model

Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views

3D Deep Shape Descriptor.

Learning Barycentric Representations of 3D Shapes for Sketch-Based 3D Shape Retrieval.

Progressive Shape-Distribution-Encoder for Learning 3D Shape Representation

Deep Spatiality: Unsupervised Learning of Spatially-Enhanced Global and Local 3D Features by Deep Neural Network With Coupled Softmax.

Progressive Shape-Distribution-Encoder for 3D Shape Retrieval

Deep Nonlinear Metric Learning for 3-D Shape Retrieval

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Deep Multimetric Learning for Shape-Based 3D Model Retrieval

Unify 3D Shape Retrieval and Classification in One Framework

Unsupervised Deep Shape Descriptor with Point Distribution Learning.

Hamming Embedding Sensitivity Guided Fusion Network for 3D Shape Representation.