Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Object Category Recognition Using Generative Template Boosting.

Object categorization with sketch representation and generalized samples

A stochastic graph grammar for compositional object representation and recognition

Learning and-or templates for object recognition by information projection

Learning And-Or Templates for Object Recognition and Detection

Learning Mixed Templates for Object Recognition

Unsupervised Learning of Stochastic AND-OR Templates for Object Modeling.

Object Category Recognition Using Boosting Tree With Heterogeneous Features

From Bag Of Categories To Tree Of Object Recognition

Compositional Boosting for Computing Hierarchical Image Structures

Learning Compositional Models for Object Categories from Small Sample Sets

Learning a Probabilistic Topology Discovering Model for Scene Categorization.

FINE-GRAINED AND LAYERED OBJECT RECOGNITION

Learning Structural Representations via Dynamic Object Landmarks Discovery for Sketch Recognition and Retrieval

Open-Ended Fine-Grained 3D Object Categorization by Combining Shape and Texture Features in Multiple Colorspaces

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

An Empirical Study of Object Category Recognition: Sequential Testing with Generalized Samples

Robust Object Recognition Via Weakly Supervised Metric and Template Learning

Dynamical And-Or Graph Learning for Object Shape Modeling and Detection

Generic object recognition with regional statistical models and layer joint boosting