Abstract:The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.

WSDM 2024 Workshop on Large Language Models for Individuals, Groups, and Society.

The Second Workshop on Large Language Models for Individuals, Groups, and Society

Shaping the Emerging Norms of Using Large Language Models in Social Computing Research

Personalization of Large Language Models: A Survey

Socially Responsible Data for Large Multilingual Language Models

Apprentices to Research Assistants: Advancing Research with Large Language Models

Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

Large Language Models for Education: A Survey and Outlook

An Interdisciplinary Outlook on Large Language Models for Scientific Research

Personalized Multimodal Large Language Models: A Survey

A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions

WavLLM: Towards Robust and Adaptive Speech Large Language Model

The Impact of Large Language Models in Academia: from Writing to Speaking

Challenges and Applications of Large Language Models

14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon

Large Language Models for Social Networks: Applications, Challenges, and Solutions

MM-LLMs: Recent Advances in MultiModal Large Language Models