Abstract:The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{<a class="link-external link-http" href="http://aka.ms/wavllm" rel="external noopener nofollow">this http URL</a>}.

UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages

Efficient Continual Pre-training of LLMs for Low-resource Languages

RedStone: Curating General, Code, Math, and QA Data for Large Language Models

Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments

Unlocking the Potential of Model Merging for Low-Resource Languages

DataComp-LM: In search of the next generation of training sets for language models

LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages

Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling

Large-scale Language Model Rescoring on Long-form Data

A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs

Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

No Language Left Behind: Scaling Human-Centered Machine Translation

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging

CROME: Cross-Modal Adapters for Efficient Multimodal LLM

Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners