Abstract:Reconstructing three-dimensional (3D) objects from images has attracted increasing attention due to its wide applications in computer vision and robotic tasks. Despite the promising progress of recent deep learning-based approaches, which directly reconstruct the full 3D shape without considering the conceptual knowledge of the object categories, existing models have limited usage and usually create unrealistic shapes. 3D objects have multiple forms of representation, such as 3D volume, conceptual knowledge, and so on. In this work, we show that the conceptual knowledge for a category of objects, which represents objects as prototype volumes and is structured by graph, can enhance the 3D reconstruction pipeline. We propose a novel multimodal framework that explicitly combines graph-based conceptual knowledge with deep neural networks for 3D shape reconstruction from a single RGB image. Our approach represents conceptual knowledge of a specific category as a structure-based knowledge graph. Specifically, conceptual knowledge acts as visual priors and spatial relationships to assist the 3D reconstruction framework to create realistic 3D shapes with enhanced details. Our 3D reconstruction framework takes an image as input. It first predicts the conceptual knowledge of the object in the image, then generates a 3D object based on the input image and the predicted conceptual knowledge. The generated 3D object satisfies the following requirements: (1) it is consistent with the predicted graph in concept, and (2) consistent with the input image in geometry. Extensive experiments on public datasets (i.e., ShapeNet, Pix3D, and Pascal3D+) with 13 object categories show that (1) our method outperforms the state-of-the-art methods, (2) our prototype volume-based conceptual knowledge representation is more effective, and (3) our pipeline-agnostic approach can enhance the reconstruction quality of various 3D shape reconstruction pipelines.

Deep-Learning Methods of Cross-Modal Tasks for Conceptual Design of Product Shapes: A Review

Neurocognition-inspired Design with Machine Learning

Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conception

Human-in-the-loop design with machine learning

A Conceptual Design Method Based on C-K Theory and Large Language Models

A cognitive analysis-based key concepts derivation approach for product design

Towards the Conceptual Design of ML-enhanced Products: the UX Value Framework and the CoMLUX Design Process

Brain-like Behavior Effect-inspiring and Principle Structure-combining Method for Mechanical Product Conceptual Design

Multi-modal Machine Learning in Engineering Design: A Review and Future Directions

DesignFusion: Integrating Generative Models for Conceptual Design Enrichment

Research on intelligent product conceptual design based on cognitive process

A Task-Decomposed AI-Aided Approach for Generative Conceptual Design

An Integrated Cognitive Computing Approach for Systematic Conceptual Design

ML Lifecycle Canvas: Designing Machine Learning-Empowered UX with Material Lifecycle Thinking

Product innovation concept generation based on deep learning and Kansei engineering

New Fashion: Personalized 3D Design with a Single Sketch Input

Machine Learning-Based Design Concept Evaluation

Attention-Enhanced Multimodal Learning for Conceptual Design Evaluations

A Tool to Facilitate the Cross-Cultural Design Process Using Deep Learning

From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

Enhanced 3D Shape Reconstruction With Knowledge Graph of Category Concept