Abstract:Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as Automatic Speech Recognition (ASR), and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (\textbf{A}udio \textbf{I}nst\textbf{R}uction \textbf{Bench}mark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: \textit{foundation} and \textit{chat} benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.

CharacterBench: Benchmarking Character Customization of Large Language Models

CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation

SocialBench: Sociality Evaluation of Role-Playing Conversational Agents

PersoBench: Benchmarking Personalized Response Generation in Large Language Models

CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models

MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues

FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback

SimulBench: Evaluating Language Models with Creative Simulation Tasks

DialogBench: Evaluating LLMs as Human-like Dialogue Systems

RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

AlignBench: Benchmarking Chinese Alignment of Large Language Models

AgentBench: Evaluating LLMs as Agents

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

HLB: Benchmarking LLMs' Humanlikeness in Language Use

A User-Centric Benchmark for Evaluating Large Language Models.

RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

Characteristic AI Agents via Large Language Models

CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

VoiceBench: Benchmarking LLM-Based Voice Assistants