Abstract:Transformers, which were originally developed for natural language processing, have recently generated significant interest in the computer vision and audio communities due to their flexibility in learning long-range relationships. Constrained by the data hungry nature of transformers and the limited amount of labelled data, most transformer-based models for audio tasks are finetuned from ImageNet pretrained models, despite the huge gap between the domain of natural images and audio. This has motivated the research in self-supervised pretraining of audio transformers, which reduces the dependency on large amounts of labeled data and focuses on extracting concise representations of audio spectrograms. In this paper, we propose \textbf{L}ocal-\textbf{G}lobal \textbf{A}udio \textbf{S}pectrogram v\textbf{I}sion \textbf{T}ransformer, namely ASiT, a novel self-supervised learning framework that captures local and global contextual information by employing group masked model learning and self-distillation. We evaluate our pretrained models on both audio and speech classification tasks, including audio event classification, keyword spotting, and speaker identification. We further conduct comprehensive ablation studies, including evaluations of different pretraining strategies. The proposed ASiT framework significantly boosts the performance on all tasks and sets a new state-of-the-art performance in five audio and speech classification tasks, outperforming recent methods, including the approaches that use additional datasets for pretraining.

An Improved Audio Classification Method Based on Parameter-Free Attention Combined with Self-Supervision

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Improving Acoustic Scene Classification Via Self-Supervised and Semi-Supervised Learning with Efficient Audio Transformer

Exploring the Power of Pure Attention Mechanisms in Blind Room Parameter Estimation

CAT: Causal Audio Transformer for Audio Classification

Audio xLSTMs: Learning Self-Supervised Audio Representations with xLSTMs

Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers

ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification

Multi-level Attention Model for Weakly Supervised Audio Classification

IMPROVING MULTIMODAL SPEECH ENHANCEMENT BY INCORPORATING SELF-SUPERVISED AND CURRICULUM LEARNING

Hand-crafted Attention is All You Need? A Study of Attention on Self-supervised Audio Transformer

WA-Transformer: Window Attention-based Transformer with Two-stage Strategy for Multi-task Audio Source Separation

Dual-Branch Attention-In-Attention Transformer for Single-Channel Speech Enhancement

Hand-crafted Attention is All You Need? A Study of Attention on Self-supervised Audio Transformer.

Improving Visual Speech Enhancement Network by Learning Audio-visual Affinity with Multi-head Attention

Self-supervised Attention Model for Weakly Labeled Audio Event Classification

Audio Classification Using Attention-Augmented Convolutional Neural Network

Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision.

Audio Mamba: Pretrained Audio State Space Model For Audio Tagging

Understanding Self-Attention of Self-Supervised Audio Transformers