Abstract:Sketch is a well-researched topic in the vision community by now. Sketch semantic segmentation in particular, serves as a fundamental step towards finer-level sketch interpretation. Recent works use various means of extracting discriminative features from sketches and have achieved considerable improvements on segmentation accuracy. Common approaches for this include attending to the sketch-image as a whole, its stroke-level representation or the sequence information embedded in it. However, they mostly focus on only a part of such multi-facet information. In this paper, we for the first time demonstrate that there is complementary information to be explored across all these three facets of sketch data, and that segmentation performance consequently benefits as a result of such exploration of sketch-specific information. Specifically, we propose the Sketch-Segformer, a transformer-based framework for sketch semantic segmentation that inherently treats sketches as stroke sequences other than pixel-maps. In particular, Sketch-Segformer introduces two types of self-attention modules having similar structures that work with different receptive fields (i.e., whole sketch or individual stroke). The order embedding is then further synergized with spatial embeddings learned from the entire sketch as well as localized stroke-level information. Extensive experiments show that our sketch-specific design is not only able to obtain state-of-the-art performance on traditional figurative sketches (such as SPG, SketchSeg-150K datasets), but also performs well on creative sketches that do not conform to conventional object semantics (CreativeSketch dataset) thanks for our usage of multi-facet sketch information. Ablation studies, visualizations, and invariance tests further justifies our design choice and the effectiveness of Sketch-Segformer. Codes are available at https://github.com/PRIS-CV/Sketch-SF.

Sketching by Perceptual Grouping.

Im2Sketch: Sketch Generation by Unconflicted Perceptual Grouping

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

Sketch-Based Retrieval in Large-Scale Image Database Via Position-Aware Silhouette Matching.

Sketch-based 3D Modeling by Aligning Outlines of an Image

Toward Deep Universal Sketch Perceptual Grouper

Automatic Image-Based Pencil Sketch Rendering

Free hand-drawn sketch segmentation

Universal Sketch Perceptual Grouping

Segmentation of Online Sketching Using Geometric Feature

Photo-to-Sketch Transformation in a Complex Background.

On-Line Sketch Recognition Using Direction Feature

Universal Perceptual Grouping

Making Better Use of Edges for Sketch Generation

Component Segmentation of Sketches Used in 3D Model Retrieval

Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches.

Image sketching using low-, mid-level vision cues

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

Designers' perception during sketching: An examination of Creative Segment theory using eye movements

Why Does the Sketch Look So Vivid?

SketchAA: Abstract Representation for Abstract Sketches