Abstract:Zero-shot sketch-based image retrieval (ZS-SBIR) has attracted great attention recently, due to the potential application of sketch-based retrieval under zero-shot scenarios, where the categories of query sketches and gallery photos are not observed in the training stage. However, it is still under insufficient exploration for the general and practical scenario when the query sketches and gallery photos contain both seen and unseen categories. Such a problem is defined as generalized zero-shot sketch-based image retrieval (GZS-SBIR), which is the focus of this work. To this end, we propose a novel Augmented Multi-modality Fusion (AMF) framework to generalize seen concepts to unobserved ones efficiently. Specifically, a novel knowledge discovery module named cross-domain augmentation is designed in both visual and semantic space to mimic novel knowledge unseen from the training stage, which is the key to handling the GZS-SBIR challenge. Moreover, a triplet domain alignment module is proposed to couple the cross-domain distribution between photo and sketch in visual space. To enhance the robustness of our model, we explore embedding propagation to refine both visual and semantic features by removing undesired noise. Eventually, visual-semantic fusion representations are concatenated for further domain discrimination and task-specific recognition, which tend to trigger the cross-domain alignment in both visual and semantic feature space. Experimental evaluations are conducted on popular ZS-SBIR benchmarks as well as a new evaluation protocol designed for GZS-SBIR from DomainNet dataset with more diverse sub-domains, and the promising results demonstrate the superiority of the proposed solution over other baselines. The source code is available at https://github.com/scottjingtt/AMF_GZS_SBIR.git.

Cross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

Generative Domain-Migration Hashing For Sketch-To-Image Retrieval

Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval

A Multi-Layer Deep Fusion Convolutional Neural Network for Sketch Based Image Retrieval

Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval

Instance-Level Coupled Subspace Learning for Fine-Grained Sketch-Based Image Retrieval.

A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Retrieval

Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax.

BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR

Cross-Domain Adversarial Feature Learning For Sketch Re-Identification

Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval

Enhancing Sketch-Based Image Retrieval Via Deep Discriminative Representation.

Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval.

Stacked Semantic-Guided Network for Zero-Shot Sketch-Based Image Retrieval.

Augmented Multimodality Fusion for Generalized Zero-Shot Sketch-Based Visual Retrieval

Relation-Aware Meta-Learning for Zero-shot Sketch-Based Image Retrieval

Dark-Aware Network for Fine-Grained Sketch-Based Image Retrieval

Towards Sketch-Based Image Retrieval with Deep Cross-Modal Correlation Learning.

ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval