Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Modified U-Net based 3D reconstruction model to estimate volume from multi-view images of a solid object

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

A Calibration-Informed Deep Learning Model for Three-Dimensional Particle Reconstruction of Volumetric Particle Image Velocimetry

3D Multiple-Contextual ROI-Attention Network for Efficient and Accurate Volumetric Medical Image Segmentation.

3D reconstruction and volume measurement of irregular objects based on RGB-D camera

Visual Odometry Based 3D-Reconstruction

Adaptive fish school search optimized resnet for multi-view 3D objects reconstruction

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept

Multi-View Stereo Representation Revist: Region-Aware MVSNet

A Global Max-Flow-Based Multi-Resolution Next-Best-View Method for Reconstruction of 3D Unknown Objects.

3D-RVP: A method for 3D object reconstruction from a single depth view using voxel and point

Reconstruction of 3D Object Shape Using Hybrid Modular Neural Network Architecture Trained on 3D Models from ShapeNetCore Dataset

SparseVoxNet: 3-D Object Recognition With Sparsely Aggregation of 3-D Dense Blocks

3D model retrieval based on volume distribution

Ground 3D Object Reconstruction Based on Multi-View 3D Occupancy Network using Satellite Remote Sensing Image

3D Reconstruction for Multi-view Objects

3D Volume Reconstruction from Single Lateral X-Ray Image via Cross-Modal Discrete Embedding Transition.

Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image

An Active Volumetric Model for 3D Reconstruction

Deep Single-View 3D Object Reconstruction with Visual Hull Embedding