Abstract:Jointly using text and sketch for scene-level image retrieval utilizes the complementary between text and sketch to describe the fine-grained scene content and retrieve the target image, which plays a pivotal role in accurate image retrieval. Existing methods directly fuse the features of sketch and text and thus suffer from the bottleneck of limited utilization for crucial semantic and structural information, leading to inaccurate matching with images. In this paper, we propose SceneDiff, a novel retrieval network that leverages a pre-trained diffusion model to establish a shared generative latent space, enabling a joint latent representation learning for both sketch and text features and precise alignment with the corresponding image. Specifically, we encode text, sketch and image features, and project them into the diffusion-based share space, conditioning the denoising process on sketch and text features to generate latent fusion features, while employing the pre-trained autoencoder for latent image features. Within this space, we introduce the content-aware feature transformation module to reconcile encoded sketch and image features with the diffusion latent space's dimensional requirements and preserve their visual content information. Then we augment the representation capability of the generated latent fusion features by integrating multiple samplings with partition attention, and utilize contrastive learning to align both direct fusion features and generated latent fusion features with corresponding image representations. Our method outperforms the state-of-the-art works through extensive experiments, providing a novel insight into the related retrieval field.

SketchScene: Scene Sketch to Image Generation with Diffusion Models.

Sketch-Guided Scene Image Generation

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

DiffSketching: Sketch Control Image Synthesis with Diffusion Models

Language-based colorization of scene sketches

SketchFFusion: Sketch-guided image editing with diffusion model

SketchKnitter: Vectorized Sketch Generation with Diffusion Models

SceneDiff: Generative Scene-Level Image Retrieval with Text and Sketch Using Diffusion Models

Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training

External Knowledge Enhanced 3D Scene Generation from Sketch

Unsupervised Scene Sketch to Photo Synthesis

Text-Guided Scene Sketch-to-Photo Synthesis

SketchyScene: Richly-Annotated Scene Sketches

Joint Generative Modeling of Scene Graphs and Images via Diffusion Models

Training-Free Sketch-Guided Diffusion with Latent Optimization

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation

SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation

S 5 : Sketch-to-image Synthesis via Scene and Size Sensing

ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text