Abstract:Conceptual design is the foundational stage of a design process that translates ill-defined design problems into low-fidelity design concepts and prototypes. While deep learning approaches are widely applied in later design stages for design automation, we see fewer attempts in conceptual design for three reasons: 1) the data in this stage exhibit multiple modalities: natural language, sketches, and 3D shapes, and these modalities are challenging to represent in deep learning methods; 2) it requires knowledge from a larger source of inspiration instead of focusing on a single design task; and 3) it requires translating designers' intent and feedback, and hence needs more interaction with humans, either designers or users. With recent advances in deep learning of cross-modal tasks (DLCMT) and the availability of large cross-modal datasets, we see opportunities to apply these learning-based methods to the conceptual design of product shapes. In this paper, we conduct a systematic review on the methods for DLCMT that involve three design modalities: natural language, sketches, and 3D shapes, which revealed 50 articles in the fields of computer graphics, computer vision, and engineering design. This review work identifies the key challenges and opportunities in applying DLCMT in the conceptual design of engineered products. The authors also propose a list of five research questions that point to future directions and call on the community to devote itself to principled research investigations that help translate knowledge from computer science to engineering design.

Deep Learning for Identifying Potential Conceptual Shifts for Co-creative Drawing

An online sketching and gesture editing system for conceptual design

Relating Cognitive Models of Design Creativity to the Similarity of Sketches Generated by an AI Partner

Sketch-based Creativity Support Tools using Deep Learning

Exploring Co-creative Drawing Workflows

SmartPaint: a Co-Creative Drawing System Based on Generative Adversarial Networks

Human-in-the-loop design with machine learning

Drawing with AI -- Exploring Collaborative Inking Experiences Based on Mid-air Pointing and Reinforcement Learning

A Collaborative, Interactive and Context-Aware Drawing Agent for Co-Creative Design

Dual Embodied-Symbolic Concept Representations for Deep Learning

Deep-Learning Methods of Cross-Modal Tasks for Conceptual Design of Product Shapes: A Review

Creativity of Deep Learning: Conceptualization and Assessment

Deep learning for studying drawing behavior: A review

Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks

Two-Dimensional Sketch Recognition with Design Intent Capture

Parallel developmental changes in children’s production and recognition of line drawings of visual concepts

Deep learning for symbols detection and classification in engineering drawings

Concept-Oriented Deep Learning

DeepDrawing: A Deep Learning Approach to Graph Drawing

Concept-Oriented Deep Learning: Generative Concept Representations

Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI