Abstract:This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL), wherein the model is trained to recognize multiple unseen classes within a sample (e.g., an image) based on seen classes and auxiliary knowledge, e.g., semantic information. Existing methods usually resort to analyzing the relationship of various seen classes residing in a sample from the dimension of spatial or semantic characteristics and transferring the learned model to unseen ones. However, they neglect the integrity of local and global features. Although the use of the attention structure will accurately locate local features, especially objects, it will significantly lose its integrity, and the relationship between classes will also be affected. Rough processing of global features will also directly affect comprehensiveness. This neglect will make the model lose its grasp of the main components of the image. Relying only on the local existence of seen classes during the inference stage introduces unavoidable bias. In this paper, we propose a novel and comprehensive visual-semantic framework for MLZSL, dubbed Epsilon, to fully make use of such properties and enable a more accurate and robust visual-semantic projection. In terms of spatial information, we achieve effective refinement by group aggregating image features into several semantic prompts. It can aggregate semantic information rather than class information, preserving the correlation between semantics. In terms of global semantics, we use global forward propagation to collect as much information as possible to ensure that semantics are not omitted. Experiments on large-scale MLZSL benchmark datasets NUS-Wide and Open-Images-v4 demonstrate that the proposed Epsilon outperforms other state-of-the-art methods with large margins.

AMS-SFE: Towards an Alignment of Manifold Structures via Semantic Feature Expansion for Zero-shot Learning

GENERATING MANIFOLD-ALIGNED SEMANTIC FEATURE FOR ZERO-SHOT LEARNING

A Novel Perspective to Zero-shot Learning: Towards an Alignment of Manifold Structures via Semantic Feature Expansion

Joint Learning of Attended Zero-Shot Features and Visual-Semantic Mapping.

Learning Visually Aligned Semantic Graph for Cross-Modal Manifold Matching.

Manifold Regularized Cross-Modal Embedding for Zero-Shot Learning

Zero-Shot Leaning With Manifold Embedding

Manifold Embedding for Zero-Shot Recognition

Dual Collaborative Visual-Semantic Mapping for Multi-Label Zero-Shot Image Recognition

Matrix Tri-Factorization With Manifold Regularizations For Zero-Shot Learning

Adaptive multi-scale semantic fusion network for zero-shot learning

Stacked Semantic-Guided Attention Model for Fine-Grained Zero-Shot Learning.

Zero-Shot Learning via Structure-Aligned Generative Adversarial Network

Zero-Shot Learning via Discriminative Dual Semantic Auto-Encoder

Semantic Autoencoder for Zero-Shot Learning

An Inverse Mapping With Manifold Alignment For Zero-Shot Learning

Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment

Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning

Zero-Shot Learning on Semantic Class Prototype Graph

Zero-Shot Learning With Attentive Region Embedding and Enhanced Semantics

Delving into Shape-aware Zero-shot Semantic Segmentation