Abstract:The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.

Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets

EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation

GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning

LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

MaLA-500: Massive Language Adaptation of Large Language Models

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities

LLM for Everyone: Representing the Underrepresented in Large Language Models

SambaLingo: Teaching Large Language Models New Languages

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic

Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasets

LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models

Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

Large Language Models for Data Annotation and Synthesis: A Survey

DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models

Extrapolating Large Language Models to Non-English by Aligning Languages

Conversations in Galician: a Large Language Model for an Underrepresented Language

LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation