Abstract:Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. Despite the success, most traditional VLMs-based methods are restricted by the assumption of partial source supervision or ideal vocabularies, which rarely satisfy the open-world scenario. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. To address this challenge, we propose the Self Structural Semantic Alignment (S^3A) framework, which extracts the structural semantic information from unlabeled data while simultaneously self-learning. Our S^3A framework adopts a unique Cluster-Vote-Prompt-Realign (CVPR) algorithm, which iteratively groups unlabeled data to derive structural semantics for pseudo-supervision. Our CVPR process includes iterative clustering on images, voting within each cluster to identify initial class candidates from the vocabulary, generating discriminative prompts with large language models to discern confusing candidates, and realigning images and the vocabulary as structural semantic alignment. Finally, we propose to self-learn the CLIP image encoder with both individual and structural semantic alignment through a teacher-student learning strategy. Our comprehensive experiments across various generic and fine-grained benchmarks demonstrate that the S^3A method offers substantial improvements over existing VLMs-based approaches, achieving a more than 15% accuracy improvement over CLIP on average. Our codes, models, and prompts are publicly released at <a class="link-external link-https" href="https://github.com/sheng-eatamath/S3A" rel="external noopener nofollow">this https URL</a>.

Dynamic VAEs Via Semantic-Aligned Matching for Continual Zero-Shot Learning

Joint Learning of Attended Zero-Shot Features and Visual-Semantic Mapping.

Dual Collaborative Visual-Semantic Mapping for Multi-Label Zero-Shot Image Recognition

A Discriminative Cross-Aligned Variational Autoencoder for Zero-Shot Learning

Zero-Shot Learning With Attentive Region Embedding and Enhanced Semantics

Continual Zero-Shot Learning through Semantically Guided Generative Random Walks

Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning

Semantic Autoencoder for Zero-Shot Learning

Zero-Shot Learning via Discriminative Dual Semantic Auto-Encoder

Semantic Softmax Loss for Zero-Shot Learning

Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment

Meta-Transfer Networks for Zero-Shot Learning

Multi-modal Generative Adversarial Network for Zero-Shot Learning

Dynamic visual-guided selection for zero-shot learning

Visual-Semantic Aligned Bidirectional Network for Zero-Shot Learning

Hydroxyethylstärke: Stellungnahme der Präsidenten der DGAI und des BDA sowie der Präsidentin der DAAF

Leveraging Dual Variational Autoencoders and Generative Adversarial Networks for Enhanced Multimodal Interaction in Zero-Shot Learning

Manifold Regularized Cross-Modal Embedding for Zero-Shot Learning

Visually Aligned Word Embeddings for Improving Zero-shot Learning.

Dual VAEGAN: A generative model for generalized zero-shot learning

Learning discriminative visual semantic embedding for zero-shot recognition