MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

Benno Weck,Ilaria Manco,Emmanouil Benetos,Elio Quinton,George Fazekas,Dmitry Bogdanov

2024-08-02

Abstract:Multimodal models that jointly process audio and language hold great promise in audio understanding and are increasingly being adopted in the music domain. By allowing users to query via text and obtain information about a given audio input, these models have the potential to enable a variety of music understanding tasks via language-based interfaces. However, their evaluation poses considerable challenges, and it remains unclear how to effectively assess their ability to correctly interpret music-related inputs with current methods. Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music understanding in multimodal language models focused on audio. MuChoMusic comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets, and covering a wide variety of genres. Questions in the benchmark are crafted to assess knowledge and reasoning abilities across several dimensions that cover fundamental musical concepts and their relation to cultural and functional contexts. Through the holistic analysis afforded by the benchmark, we evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality, pointing to a need for better multimodal integration. Data and code are open-sourced.

Sound,Computation and Language,Machine Learning,Multimedia,Audio and Speech Processing

What problem does this paper attempt to address?

The paper primarily addresses the issue of evaluating music comprehension capabilities in multimodal audio-language models. Specifically, the paper introduces the MuChoMusic benchmark dataset, which aims to assess these models' understanding of music through a series of carefully designed multiple-choice questions. The MuChoMusic benchmark dataset contains 1,187 manually verified multiple-choice questions, covering 644 tracks from two public music datasets and involving various music genres. These questions are designed to test the models' knowledge and reasoning abilities across multiple dimensions, including music theory, stylistic traditions, historical and social contexts, among others. By comprehensively analyzing these dimensions, the models' music comprehension capabilities can be thoroughly evaluated. The paper also details the construction process of the MuChoMusic benchmark dataset, including how multiple-choice questions are automatically generated from human-written music descriptions and how these questions are manually annotated for verification. Additionally, the paper compares MuChoMusic with other related benchmark datasets and outlines the evaluation metrics and experimental methods used. In summary, the main objective of this research is to provide a standardized evaluation framework to objectively measure and improve the understanding and application capabilities of multimodal audio-language models in the field of music.

MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music

OpenMU: Your Swiss Army Knife for Music Understanding

MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models

The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models

Evaluation of pretrained language models on music understanding

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning

LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment

MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks

WikiMuTe: A web-sourced dataset of semantic descriptions for music audio

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation

M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation

Multimodal music datasets? Challenges and future goals in music processing

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria