Abstract:Automated Audio Captioning (AAC) involves generating natural language descriptions of audio content, using encoder-decoder architectures. An audio encoder produces audio embeddings fed to a decoder, usually a Transformer decoder, for caption generation. In this work, we describe our model, which novelty, compared to existing models, lies in the use of a ConvNeXt architecture as audio encoder, adapted from the vision domain to audio classification. This model, called CNext-trans, achieved state-of-the-art scores on the AudioCaps (AC) dataset and performed competitively on Clotho (CL), while using four to forty times fewer parameters than existing models. We examine potential biases in the AC dataset due to its origin from AudioSet by investigating unbiased encoder's impact on performance. Using the well-known PANN's CNN14, for instance, as an unbiased encoder, we observed a 1.7% absolute reduction in SPIDEr score (where higher scores indicate better performance). To improve cross-dataset performance, we conducted experiments by combining multiple AAC datasets (AC, CL, MACS, WavCaps) for training. Although this strategy enhanced overall model performance across datasets, it still fell short compared to models trained specifically on a single target dataset, indicating the absence of a one-size-fits-all model. To mitigate performance gaps between datasets, we introduced a Task Embedding (TE) token, allowing the model to identify the source dataset for each input sample. We provide insights into the impact of these TEs on both the form (words) and content (sound event types) of the generated captions. The resulting model, named CoNeTTE, an unbiased CNext-trans model enriched with dataset-specific Task Embeddings, achieved SPIDEr scores of 44.1% and 30.5% on AC and CL, respectively. Code available: <a class="link-external link-https" href="https://github.com/Labbeti/conette-audio-captioning" rel="external noopener nofollow">this https URL</a>.

CoTel: Ontology-Neural Co-Enhanced Text Labeling

Citation Metadata Extraction Via Deep Neural Network-based Segment Sequence Labeling

Text Categorization Based on Domain Ontology

CoTea: Collaborative teaching for low-resource named entity recognition with a divide-and-conquer strategy

TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding

COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval

CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition.

Rethinking Crowdsourcing Annotation: Partial Annotation with Salient Labels for Multi-Label Image Classification

CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases

ComCo: Complementary supervised contrastive learning for complementary label learning

LED: Label Correlation Enhanced Decoder for Multi-Label Text Classification

VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool

CoSQA: 20,000+ Web Queries for Code Search and Question Answering

CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding

Empowering Biomedical Named Entity Recognition Through Multi-Tagger Collaboration

KeNet:Knowledge-enhanced Doc-Label Attention Network for Multi-label text classification

CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification

xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning

LocTex: Learning Data-Efficient Visual Representations from Localized Textual Supervision

DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations