Abstract:Audio-visual zero-shot learning aims to recognize unseen classes based on paired audio-visual sequences. Recent methods mainly focus on learning multi-modal features aligned with class names to enhance the generalization ability to unseen categories. However, these approaches ignore the obscure event concepts in class names and may inevitably introduce complex network structures with difficult training objectives. In this paper, we introduce a straightforward yet efficient framework called KnowleDge-Augmented audio-visual learning (KDA), which aids the model in more effectively learning novel event content by leveraging an external knowledge base. Specifically, we first propose to utilize the knowledge contained in large language models (LLMs) to generate numerous descriptive sentences that include important distinguishing audio-visual features of event classes, which helps to better understand unseen categories. Furthermore, we propose a knowledge-aware adaptive margin loss to help distinguish similar events, further improving the generalization ability towards unseen classes. Extensive experimental results demonstrate that our proposed KDA can outperform state-of-the-art methods on three popular audio-visual zero-shot learning datasets.Our code will be avaliable at \url{<a class="link-external link-https" href="https://github.com/chenhaoxing/KDA" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively recognize unseen classes in audio - visual zero - shot learning (AVZSL). Existing methods mainly focus on learning the alignment of multimodal features with class names to enhance the generalization ability for unseen classes. However, these methods overlook the ambiguous event concepts in class names and may introduce complex network structures and difficult - to - train objectives. This paper proposes a new framework - Knowledge - Augmented audio - visual learning (KDA), which helps the model learn new event content more effectively by using external knowledge bases. ### Specific Problems and Solutions 1. **Problem**: When dealing with unseen classes, existing methods are difficult to make accurate predictions due to the lack of necessary knowledge and context. - **Solution**: The KDA framework uses large - language models (LLMs) to generate descriptive sentences containing important discriminative audio - visual features, which helps to better understand unseen classes. 2. **Problem**: Directly mapping audio - visual features to the label feature space is not efficient and may introduce understanding biases. - **Solution**: The KDA framework maps audio - visual features and textual knowledge features to a common space and uses alignment loss to ensure that samples within the same class are as close as possible to their corresponding event descriptions, thereby improving the generalization ability of the model. 3. **Problem**: The discrimination between similar events is not high, which affects the classification performance of the model. - **Solution**: The KDA framework proposes a knowledge - aware adaptive margin loss. By adding an adaptive margin to the classification loss, different classes are effectively separated. This adaptive margin is generated based on the knowledge similarity between each pair of classes. ### Main Contributions 1. **Proposed a new audio - visual zero - shot learning framework**: By using the knowledge of large - language models, the generalization ability for unseen event classes is significantly improved. 2. **Proposed a knowledge - aware adaptive margin loss**: Further separates different classes in the common embedding space according to description similarity. 3. **Extensive experimental verification**: Experimental results show that the proposed KDA framework outperforms existing models on three popular audio - visual zero - shot learning datasets. A detailed analysis shows the advantages of the proposed model architecture. ### Experimental Setup - **Datasets**: VGGSound - GZSL, UCF - GZSL, ActivityNet - GZSL. - **Feature Extraction**: Use self - supervised SeLaVi to obtain main features, and use C3D and VGGish to obtain classification features. - **Training Details**: Use the Adam optimizer and set different hyper - parameters, such as learning rate, batch size, dropout ratio, etc. - **Evaluation Metrics**: Use S, U, ZSL and HM (harmonic mean) to evaluate the performance of the model on seen and unseen classes. ### Conclusion This paper significantly improves the ability to recognize unseen classes in audio - visual zero - shot learning tasks by introducing the Knowledge - Augmented audio - visual learning framework (KDA). By using large - language models to generate detailed event descriptions and combining with the knowledge - aware adaptive margin loss, the KDA framework achieves excellent performance on multiple datasets.

Boosting Audio-visual Zero-shot Learning with Large Language Models

Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language

Enhancing Zero-shot Audio Classification using Sound Attribute Knowledge from Large Language Models

Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

Audio-visual Generalized Zero-shot Learning the Easy Way

Object-Aware Image Augmentation for Audio-Visual Zero-Shot Learning

Out-Of-Distribution Detection for Audio-visual Generalized Zero-Shot Learning: A General Framework

K-LITE: Learning Transferable Visual Models with External Knowledge

Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

Modality-Fusion Spiking Transformer Network for Audio-Visual Zero-Shot Learning

Audio-Visual LLM for Video Understanding

Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration

Zero-Shot Learning Over Large Output Spaces : Utilizing Indirect Knowledge Extraction from Large Language Models

MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge

Learning Visual-and-semantic Knowledge Embedding for Zero-Shot Image Classification

Knowledge-Experience Graph with Denoising Autoencoder for Zero-Shot Learning in Visual Cognitive Development.

HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning

Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning

Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition

Class knowledge overlay to visual feature learning for zero-shot image classification