Sketch-based Creativity Support Tools using Deep Learning

Forrest Huang,Eldon Schoop,David Ha,Jeffrey Nichols,John Canny

DOI: https://doi.org/10.1007/978-3-030-82681-9

2021-11-19

Abstract:Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines' ability in understanding and generating visual content. An exciting area of development explores deep-learning approaches used to model human sketches, opening opportunities for creative applications. This chapter describes three fundamental steps in developing deep-learning-driven creativity support tools that consumes and generates sketches: 1) a data collection effort that generated a new paired dataset between sketches and mobile user interfaces; 2) a sketch-based user interface retrieval system adapted from state-of-the-art computer vision techniques; and, 3) a conversational sketching system that supports the novel interaction of a natural-language-based sketch/critique authoring process. In this chapter, we survey relevant prior work in both the deep-learning and human-computer-interaction communities, document the data collection process and the systems' architectures in detail, present qualitative and quantitative results, and paint the landscape of several future research directions in this exciting area.

Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on developing creative - support tools based on deep learning. These tools are able to process and generate sketches to assist the creative process. Specifically, the paper describes three fundamental steps to achieve this goal:

1. **Data collection**: A new paired dataset was created, which contains sketches generated by UI/UX designers and mobile user interface designs. This step is crucial for training deep - learning models to understand the relationship between sketches and actual user interfaces.

2. **Sketch - based user interface retrieval system**: A method was adapted from existing computer vision techniques, and a system was developed that can retrieve relevant user interface designs based on the sketches provided by users. This system can help designers quickly find design solutions similar to the sketches, thus accelerating the design process.

3. **Conversational sketch system**: A new system that supports the natural - language - based sketch creation and criticism process was developed. Through this interaction method, users can guide the system to create or modify sketches through natural language, providing a novel way of creative communication.

These three steps together form a framework aimed at using deep - learning techniques to improve the efficiency and quality of creative work, especially in areas such as artistic creation and user interface design. The paper also reviews relevant previous work, details the data - collection process and system architecture, and presents qualitative and quantitative results, while also exploring directions for future research.