Abstract:Zero-shot sketch-based image retrieval (ZS-SBIR) has attracted great attention recently, due to the potential application of sketch-based retrieval under zero-shot scenarios, where the categories of query sketches and gallery photos are not observed in the training stage. However, it is still under insufficient exploration for the general and practical scenario when the query sketches and gallery photos contain both seen and unseen categories. Such a problem is defined as generalized zero-shot sketch-based image retrieval (GZS-SBIR), which is the focus of this work. To this end, we propose a novel Augmented Multi-modality Fusion (AMF) framework to generalize seen concepts to unobserved ones efficiently. Specifically, a novel knowledge discovery module named cross-domain augmentation is designed in both visual and semantic space to mimic novel knowledge unseen from the training stage, which is the key to handling the GZS-SBIR challenge. Moreover, a triplet domain alignment module is proposed to couple the cross-domain distribution between photo and sketch in visual space. To enhance the robustness of our model, we explore embedding propagation to refine both visual and semantic features by removing undesired noise. Eventually, visual-semantic fusion representations are concatenated for further domain discrimination and task-specific recognition, which tend to trigger the cross-domain alignment in both visual and semantic feature space. Experimental evaluations are conducted on popular ZS-SBIR benchmarks as well as a new evaluation protocol designed for GZS-SBIR from DomainNet dataset with more diverse sub-domains, and the promising results demonstrate the superiority of the proposed solution over other baselines. The source code is available at https://github.com/scottjingtt/AMF_GZS_SBIR.git.

Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval.

Zero-shot Sketch-based Image Retrieval with Adaptive Balanced Discriminability and Generalizability.

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network

Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval

Stacked Semantic-Guided Network for Zero-Shot Sketch-Based Image Retrieval.

SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches

Zero-shot sketch-based image retrieval via adaptive relation-aware metric learning

Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval

Domain Adaptation Meets Zero-Shot Learning: an Annotation-Efficient Approach to Multi-Modality Medical Image Segmentation

Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style

Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval

Relation-Aware Meta-Learning for Zero-shot Sketch-Based Image Retrieval

Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval

Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval

Zero-Shot Sketch-Image Hashing.

Zero-Shot Sketch-Based Image Retrieval with Structure-aware Asymmetric Disentanglement

Indicative Vision Transformer for end-to-end zero-shot sketch-based image retrieval

Contour detection network for zero-shot sketch-based image retrieval

Augmented Multimodality Fusion for Generalized Zero-Shot Sketch-Based Visual Retrieval

Ocean: A Dual Learning Approach for Generalized Zero-Shot Sketch-Based Image Retrieval.