Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

SliceNet: A Proficient Model for Real-Time 3D Shape-Based Recognition

Slicenet: Slice-Wise 3D Shapes Reconstruction from Single Image

Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction

SparseVoxNet: 3-D Object Recognition With Sparsely Aggregation of 3-D Dense Blocks

Reconstructing Recognizable 3D Face Shapes Based on 3D Morphable Models.

Multi-view Moments Embedding Network for 3D Shape Recognition

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

LIMAN: Local Information based Multi Attention Network for 3D Shape Recognition

Efficient Slice Anomaly Detection Network for 3D Brain MRI Volume

Latent-MVCNN: 3D Shape Recognition Using Multiple Views from Pre-defined or Random Viewpoints

Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve

MANet: Multimodal Attention Network based Point- View fusion for 3D Shape Recognition

PVRNet: Point-View Relation Neural Network for 3D Shape Recognition

Multi-view Convolutional Neural Networks for 3D Shape Recognition

DAN: Deep-Attention Network for 3D Shape Recognition

Three-Dimensional-Slice-Super-Resolution-Net: A Fast Few Shooting Learning Model for 3D Super-Resolution Using Slice-Up and Slice-Reconstruction

Learning Local Neighboring Structure for Robust 3D Shape Representation

PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition.

MInet: A Novel Network Model for Point Cloud Processing by Integrating Multi-Modal Information

MeshNet: Mesh Neural Network for 3D Shape Representation

DTV-CNN: Neural network based on depth and thickness views for efficient 3D shape classification