Abstract:Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine-grained and local editing (e.g., shape, pose, and details). In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture. This motivates us to explore a novel task of sketch concept extraction: given one or more sketch-image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch-based image synthesis and editing at a fine-grained level. To accomplish this, we introduce CustomSketching, a two-stage framework for extracting novel sketch concepts. Considering that an object can often be depicted by a contour for general shapes and additional strokes for internal details, we introduce a dual-sketch representation to reduce the inherent ambiguity in sketch depiction. We employ a shape loss and a regularization loss to balance fidelity and editability during optimization. Through extensive experiments, a user study, and several applications, we show our method is effective and superior to the adapted baselines.

InSide: Interactive Sketching for Image Database Exploration

Sketch-Based Retrieval in Large-Scale Image Database Via Position-Aware Silhouette Matching.

Rst-Invariant Sketch Retrieval Based On Circular Description

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

Model-Guided 3D Sketching

Sketch-based 3D Modeling by Aligning Outlines of an Image

SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies.

SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Sketch-based 3-D modeling for piecewise planar objects in single images

On-Line Sketch Recognition Using Direction Feature

Towards Interactive Image Inpainting via Sketch Refinement

A 3d Sketching Interacting Tool For Physical Simulation Based On Web

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

A Suggestive Interface for Sketch-Based Character Posing

A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Retrieval

CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

Edgel Index for Large-Scale Sketch-Based Image Search

Sketch Me That Shoe

IdeaPanel: A Large Scale Interactive Sketch-based Image Search System

Guess what you draw: interactive contour-based image retrieval on a million-scale database.