Abstract:Artificial Intelligence Generated Content (AIGC) has shown remarkable progress in generating realistic images. However, in this paper, we take a step "backward" and address AIGC for the most rudimentary visual modality of human sketches. Our objective is on the creative nature of sketches, and that creative sketching should take the form of an interactive process. We further enable text to drive the sketch ideation process, allowing creativity to be freely defined, while simultaneously tackling the challenge of "I can't sketch". We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images. Our proposed approach, referred to as SketchDreamer, integrates a differentiable rasteriser of Bezier curves that optimises an initial input to distil abstract semantic knowledge from a pretrained diffusion model. We utilise Score Distillation Sampling to learn a sketch that aligns with a given caption, which importantly enable both text and sketch to interact with the ideation process. Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard by expanding the text prompt while making minor adjustments to the sketch input. Through this work, we hope to aspire the way we create visual content, democratise the creative process, and inspire further research in enhancing human creativity in AIGC. The code is available at \url{<a class="link-external link-https" href="https://github.com/WinKawaks/SketchDreamer" rel="external noopener nofollow">this https URL</a>}.

Relating Cognitive Models of Design Creativity to the Similarity of Sketches Generated by an AI Partner

Sketching Feature-Based Modeling by Capturing Design Intention

Exploring Optimal Combinations: the Impact of Sequential Multimodal Inspirational Stimuli in Design Concepts on Creativity.

The Role of Mental Models in Collaborative Sketching

Collaborative Sketching in Crowdsourcing Design: A New Method for Idea Generation

The effect of AI-based inspiration on human design ideation

SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies.

It Is Your Turn: Collaborative Ideation With a Co-Creative Robot through Sketch

An Artificial Intelligence Based Data-Driven Approach for Design Ideation

Melting Your Models: An Integrated AI-based Creativity Support Tool for Inspiration Evolution

Sketch-based Creativity Support Tools using Deep Learning

Creative Segment: A descriptive theory applied to computer-aided sketching

A Comparison Study of Human and Machine-Generated Creativity

A Hybrid Prototype Method Combining Physical Models and Generative Artificial Intelligence to Support Creativity in Conceptual Design

A Collaborative, Interactive and Context-Aware Drawing Agent for Co-Creative Design

基于创意拐点的计算机辅助草图设计技术

Exploring the efficacy of hybrid-sketch: an investigation into conceptual support in the early design stage

SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation

Deep Learning for Identifying Potential Conceptual Shifts for Co-creative Drawing

Sketch Vision: Artificial Intelligence with Sight for Imagination

Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI