Abstract:Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as Automatic Speech Recognition (ASR), and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (\textbf{A}udio \textbf{I}nst\textbf{R}uction \textbf{Bench}mark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: \textit{foundation} and \textit{chat} benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to address the deficiencies in current evaluation benchmarks for large - scale audio - language models (LALMs). Specifically, existing evaluation benchmarks mainly focus on specific basic tasks, such as automatic speech recognition (ASR), and lack a comprehensive assessment of open - ended generation capabilities. This has led to the following problems: 1. **Lack of standardized evaluation benchmarks**: Existing evaluations of LALMs mainly rely on showing examples or releasing chat models for public experimentation, making it difficult to conduct fair and objective comparisons. 2. **Inability to comprehensively evaluate model capabilities**: Existing benchmarks fail to cover multiple types of audio signals (such as human speech, natural sounds, and music) and have limitations in instruction - following capabilities. 3. **Difficulty in revealing the existing limitations of models**: Due to the lack of comprehensive evaluation, it is difficult to identify the shortcomings of existing LALMs in audio understanding and instruction - following, which hinders progress in the field. To solve these problems, the author introduced **AIR - Bench** (Audio InstRuction Benchmark), a new benchmarking framework for evaluating the ability of LALMs to understand various audio signals (including human speech, natural sounds, and music) and to follow instructions for text - based interaction with humans. AIR - Bench has the following features: 1. **Comprehensive coverage of audio signals**: It covers multiple audio types, ensuring a comprehensive evaluation of LALMs' capabilities. 2. **Hierarchical benchmark structure**: It includes two dimensions, namely the basic benchmark and the chat benchmark. The basic benchmark contains approximately 19,000 single - choice questions in 19 different audio tasks, which are used to evaluate basic task capabilities; the chat benchmark contains more than 2,000 open - ended question - and - answer data, directly evaluating the model's understanding of complex audio and its instruction - following capabilities. 3. **A unified, objective, and reproducible evaluation framework**: It requires the model to directly generate hypotheses and uses advanced language models such as GPT - 4 to evaluate the quality of the generated hypotheses based on the meta - information of the audio, ensuring the consistency and credibility of the evaluation results. Through these improvements, AIR - Bench can more comprehensively evaluate the capabilities of LALMs, reveal the limitations of existing models, and provide directions for future research.

AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

AudioBench: A Universal Benchmark for Audio Large Language Models

Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models

VoiceBench: Benchmarking LLM-Based Voice Assistants

AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

AGIBench: A Multi-granularity, Multimodal, Human-referenced, Auto-scoring Benchmark for Large Language Models

SimulBench: Evaluating Language Models with Creative Simulation Tasks

Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

LLF-Bench: Benchmark for Interactive Learning from Language Feedback

ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models

A-Bench: Are LMMs Masters at Evaluating AI-generated Images?

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

DialogBench: Evaluating LLMs as Human-like Dialogue Systems

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

A Survey on Multimodal Benchmarks: In the Era of Large AI Models

RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues

DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks