Abstract:Recent general-purpose audio representations show state-of-the-art performance on various audio tasks. These representations are pre-trained by self-supervised learning methods that create training signals from the input. For example, typical audio contrastive learning uses temporal relationships among input sounds to create training signals, whereas some methods use a difference among input views created by data augmentations. However, these training signals do not provide information derived from the intact input sound, which we think is suboptimal for learning representation that describes the input as it is. In this paper, we seek to learn audio representations from the input itself as supervision using a pretext task of auto-encoding of masked spectrogram patches, Masked Spectrogram Modeling (MSM, a variant of Masked Image Modeling applied to audio spectrogram). To implement MSM, we use Masked Autoencoders (MAE), an image self-supervised learning method. MAE learns to efficiently encode the small number of visible patches into latent representations to carry essential information for reconstructing a large number of masked patches. While training, MAE minimizes the reconstruction error, which uses the input as training signal, consequently achieving our goal. We conducted experiments on our MSM using MAE (MSM-MAE) models under the evaluation benchmark of the HEAR 2021 NeurIPS Challenge. Our MSM-MAE models outperformed the HEAR 2021 Challenge results on seven out of 15 tasks (e.g., accuracies of 73.4% on CREMA-D and 85.8% on LibriCount), while showing top performance on other tasks where specialized models perform better. We also investigate how the design choices of MSM-MAE impact the performance and conduct qualitative analysis of visualization outcomes to gain an understanding of learned representations. We make our code available online.

MFAE: Masked frame-level autoencoder with hybrid-supervision for low-resource music transcription

Unaligned Supervision For Automatic Music Transcription in The Wild

DAFE-MSGAT: Dual-Attention Feature Extraction and Multi-Scale Graph Attention Network for Polyphonic Piano Transcription

A Data-Driven Analysis of Robust Automatic Piano Transcription

Annotation-free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion

High Resolution Guitar Transcription via Domain Adaptation

MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers

High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times

Improved Architecture for High-resolution Piano Transcription to Efficiently Capture Acoustic Characteristics of Music Signals

Multi-Instrument Polyphonic Melody Transcription Based on Deep Learning

MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage

YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation

Machine Learning Techniques in Automatic Music Transcription: A Systematic Survey

Semi-Supervised Convolutive NMF for Automatic Piano Transcription

Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription

Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing

Invariances and Data Augmentation for Supervised Music Transcription

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

Improving Automatic Piano Transcription by Refined Feature Fusion and Weighted Loss

Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation