Abstract:Sketch is a well-researched topic in the vision community by now. Sketch semantic segmentation in particular, serves as a fundamental step towards finer-level sketch interpretation. Recent works use various means of extracting discriminative features from sketches and have achieved considerable improvements on segmentation accuracy. Common approaches for this include attending to the sketch-image as a whole, its stroke-level representation or the sequence information embedded in it. However, they mostly focus on only a part of such multi-facet information. In this paper, we for the first time demonstrate that there is complementary information to be explored across all these three facets of sketch data, and that segmentation performance consequently benefits as a result of such exploration of sketch-specific information. Specifically, we propose the Sketch-Segformer, a transformer-based framework for sketch semantic segmentation that inherently treats sketches as stroke sequences other than pixel-maps. In particular, Sketch-Segformer introduces two types of self-attention modules having similar structures that work with different receptive fields (i.e., whole sketch or individual stroke). The order embedding is then further synergized with spatial embeddings learned from the entire sketch as well as localized stroke-level information. Extensive experiments show that our sketch-specific design is not only able to obtain state-of-the-art performance on traditional figurative sketches (such as SPG, SketchSeg-150K datasets), but also performs well on creative sketches that do not conform to conventional object semantics (CreativeSketch dataset) thanks for our usage of multi-facet sketch information. Ablation studies, visualizations, and invariance tests further justifies our design choice and the effectiveness of Sketch-Segformer. Codes are available at https://github.com/PRIS-CV/Sketch-SF.

Creative Segment: A descriptive theory applied to computer-aided sketching

Designers' perception during sketching: An examination of Creative Segment theory using eye movements

Research and Implementation of the Sketching Design System

Research on Technologies of Computer Aided Sketching Design

Sketching Feature-Based Modeling by Capturing Design Intention

Computer-Aided Sketching for 3D Conceptual configuration of Product: State-of-the-Art

Collaborative Sketching in Crowdsourcing Design: A New Method for Idea Generation

基于创意拐点的计算机辅助草图设计技术

Exploring Optimal Combinations: the Impact of Sequential Multimodal Inspirational Stimuli in Design Concepts on Creativity.

Model-Guided 3D Sketching

SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies.

The Role of Sketching States in the Stimulation of Idea Generation: an Eye Movement Study

The function of text description in sketch promotion

Segmentation of Online Sketching Using Geometric Feature

CreativeSeg: Semantic Segmentation of Creative Sketches

The Role of Mental Models in Collaborative Sketching

A Survey and Prospects of Computer Aided Sketch Technology

Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches.

Impact of text on idea generation: an electroencephalography study

Relating Cognitive Models of Design Creativity to the Similarity of Sketches Generated by an AI Partner

SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation