Abstract:Recent advancements in multimodal large language models (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concurrently process and reason about multiple modalities remains inadequately explored, partly due to the lack of comprehensive modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define models capable of such tri-modal processing as omni-language models (OLMs). OmniBench is distinguished by high-quality human annotations, ensuring that accurate responses require integrated understanding and reasoning across all three modalities. Our main findings reveal that: i) most OLMs exhibit critical limitations in instruction-following and reasoning capabilities within tri-modal contexts; and ii) most baselines models perform poorly (below 50\% accuracy) even when provided with alternative textual representations of images or/and audio. These results suggest that the ability to construct a consistent context from text, image, and audio is often overlooked in existing MLLM training paradigms. To address this gap, we curate an instruction tuning dataset of 84.5K training samples, OmniInstruct, for training OLMs to adapt to multimodal contexts. We advocate for future research to focus on developing more robust tri-modal integration techniques and training strategies to enhance OLM performance across diverse modalities. The codes and live leaderboard could be found at <a class="link-external link-https" href="https://m-a-p.ai/OmniBench" rel="external noopener nofollow">this https URL</a>.

AP16-OL7: A Multilingual Database for Oriental Languages and a Language Recognition Baseline.

AP17-OLR Challenge: Data, Plan, and Baseline

A ug 2 01 9 AP 19-OLR Challenge : Three Tasks and Their Baselines

AP18-OLR Challenge: Three Tasks and Their Baselines

AP20-OLR Challenge: Three Tasks and Their Baselines

Ju l 2 01 9 AP 19-OLR Challenge : Three Tasks and Their Baselines

AP19-OLR Challenge: Three Tasks and Their Baselines

OLR 2021 Challenge: Datasets, Rules and Baselines

C L ] 2 3 Ju l 2 02 1 OLR 2021 CHALLENGE : DATASETS , RULES AND BASELINES

The XMUSPEECH System for the AP19-OLR Challenge

Oriental Language Recognition (OLR) 2020: Summary and Analysis

OC16-CE80: A Chinese-English Mixlingual Database and A Speech Recognition Baseline

Oriental Language Recognition (OLR) 2021: Summary and Analysis

End-to-end Oriental Language Speech Recognition with Integrated Language Identification

Phone-Aware Multi-task Learning and Length Expanding for Short-Duration Language Recognition.

An undergraduate Mandarin speech database for speaker recognition research

OmniBench: Towards The Future of Universal Omni-Language Models

XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages

ORCA: A Challenging Benchmark for Arabic Language Understanding

OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset

OLISIA: a Cascade System for Spoken Dialogue State Tracking