Abstract:In this work, we investigate the problem of sketch-based object localization on natural images, where given a crude hand-drawn sketch of an object, the goal is to localize all the instances of the same object on the target image. This problem proves difficult due to the abstract nature of hand-drawn sketches, variations in the style and quality of sketches, and the large domain gap existing between the sketches and the natural images. To mitigate these challenges, existing works proposed attention-based frameworks to incorporate query information into the image features. However, in these works, the query features are incorporated after the image features have already been independently learned, leading to inadequate alignment. In contrast, we propose a sketch-guided vision transformer encoder that uses cross-attention after each block of the transformer-based image encoder to learn query-conditioned image features leading to stronger alignment with the query sketch. Further, at the output of the decoder, the object and the sketch features are refined to bring the representation of relevant objects closer to the sketch query and thereby improve the localization. The proposed model also generalizes to the object categories not seen during training, as the target image features learned by our method are query-aware. Our localization framework can also utilize multiple sketch queries via a trainable novel sketch fusion strategy. The model is evaluated on the images from the public object detection benchmark, namely MS-COCO, using the sketch queries from QuickDraw! and Sketchy datasets. Compared with existing localization methods, the proposed approach gives a $6.6\%$ and $8.0\%$ improvement in mAP for seen objects using sketch queries from QuickDraw! and Sketchy datasets, respectively, and a $12.2\%$ improvement in AP@50 for large objects that are `unseen' during training.

Query Adaptive Multiview Object Instance Search and Localization Using Sketches.

Sketch-Based Retrieval in Large-Scale Image Database Via Position-Aware Silhouette Matching.

Query Adaptive Instance Search using Object Sketches

Viewpoint-Aware Representation for Sketch-Based 3D Model Retrieval

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

Sketch-Based 3D Model Retrieval via Multi-feature Fusion

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Extended Large Scale Sketch-Based 3D Shape Retrieval.

Model-driven sketch reconstruction with structure-oriented retrieval

Freeview Sketching: View-Aware Fine-Grained Sketch-Based Image Retrieval

Sketch-Guided Object Localization in Natural Images

Query-Adaptive Small Object Search Using Object Proposals and Shape-Aware Descriptors

LiveSketch: Query Perturbations for Guided Sketch-based Visual Search

Multimodal Query-guided Object Localization

Sketch-based Image Retrieval on a Large Scale Database

Query-by-sketch Semantic Object Image Retrieval

Indexing billions of images for sketch-based retrieval.

Sketch-based 3D Model Retrieval Utilizing Adaptive View Clustering and Semantic Information

A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Retrieval

Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch

A New Sketch-Based 3D Model Retrieval Approach by Using Global and Local Features