CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

Chufeng Xiao,Hongbo Fu

2024-02-27

Abstract:Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine-grained and local editing (e.g., shape, pose, and details). In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture. This motivates us to explore a novel task of sketch concept extraction: given one or more sketch-image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch-based image synthesis and editing at a fine-grained level. To accomplish this, we introduce CustomSketching, a two-stage framework for extracting novel sketch concepts. Considering that an object can often be depicted by a contour for general shapes and additional strokes for internal details, we introduce a dual-sketch representation to reduce the inherent ambiguity in sketch depiction. We employ a shape loss and a regularization loss to balance fidelity and editability during optimization. Through extensive experiments, a user study, and several applications, we show our method is effective and superior to the adapted baselines.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

### The Problem This Paper Attempts to Solve

This paper aims to address the limitations of existing text-to-image (T2I) personalization techniques in generating new concepts. Specifically:

1. **Lack of Fine-Grained Control**: Existing personalization methods primarily rely on text descriptions, resulting in users being unable to perform fine-grained and local edits (e.g., shape, pose, and details) when generating customized images.

2. **Inaccurate Spatial Feature Capture**: Current methods overly depend on text descriptions during the image generation process, failing to accurately capture the spatial features (geometry and appearance) of the target object.

To overcome these issues, the paper proposes a new task—Sketch Concept Extraction, which involves extracting a specific sketch concept from one or more sketch-image pairs to enable sketch-based image synthesis and editing. To this end, the authors propose the CustomSketching framework, which includes a two-stage optimization process and introduces a dual-sketch representation to distinguish between shape lines and detail lines, thereby reducing the inherent ambiguity in sketch depiction. Extensive experiments and user studies demonstrate the effectiveness and superiority of this method.