Abstract:Sketch-based image retrieval (SBIR) is a challenging task due to the large cross-domain gap between sketches and natural images. How to align abstract sketches and natural images into a common high-level semantic space remains a key problem in SBIR. In this paper, we propose a novel semi-heterogeneous three-way joint embedding network (Semi3-Net), which integrates three branches (a sketch branch, a natural image branch, and an edgemap branch) to learn more discriminative cross-domain feature representations for the SBIR task. The key insight lies with how we cultivate the mutual and subtle relationships amongst the sketches, natural images, and edgemaps. A semi-heterogeneous feature mapping is designed to extract bottom features from each domain, where the sketch and edgemap branches are shared while the natural image branch is heterogeneous to the other branches. In addition, a joint semantic embedding is introduced to embed the features from different domains into a common high-level semantic space, where all of the three branches are shared. To further capture informative features common to both natural images and the corresponding edgemaps, a co-attention model is introduced to conduct common channel-wise feature recalibration between different domains. A hybrid-loss mechanism is designed to align the three branches, where an alignment loss and a sketch-edgemap contrastive loss are presented to encourage the network to learn invariant cross-domain representations. Experimental results on two widely used category-level datasets (Sketchy and TU-Berlin Extension) demonstrate that the proposed method outperforms state-of-the-art methods.

Towards Sketch-Based Image Retrieval with Deep Cross-Modal Correlation Learning.

Deep Cascaded Cross-Modal Correlation Learning for Fine-Grained Sketch-Based Image Retrieval.

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Deep Multimodal Embedding Model for Fine-grained Sketch-based Image Retrieval

Enhancing Sketch-Based Image Retrieval Via Deep Discriminative Representation.

Sketch-Based Retrieval in Large-Scale Image Database Via Position-Aware Silhouette Matching.

A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Retrieval

Sketch-based Image Retrieval with Deep Visual Semantic Descriptor

Sketch-Based 3D Model Retrieval via Multi-feature Fusion

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

A Multi-Layer Deep Fusion Convolutional Neural Network for Sketch Based Image Retrieval

Instance-Level Coupled Subspace Learning for Fine-Grained Sketch-Based Image Retrieval.

Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval

Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval

Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval

Hybrid Deep Neural Network-Based Cross-Modal Image and Text Retrieval Method for Large-Scale Data.

Single Color Sketch-Based Image Retrieval in HSV Color Space

Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval

JFLN: Joint Feature Learning Network for 2D Sketch Based 3D Shape Retrieval

Zero-shot sketch-based image retrieval via adaptive relation-aware metric learning