FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

Keyu An,Qian Chen,Chong Deng,Zhihao Du,Changfeng Gao,Zhifu Gao,Yue Gu,Ting He,Hangrui Hu,Kai Hu,Shengpeng Ji,Yabin Li,Zerui Li,Heng Lu,Haoneng Luo,Xiang Lv,Bin Ma,Ziyang Ma,Chongjia Ni,Changhe Song,Jiaqi Shi,Xian Shi,Hao Wang,Wen Wang,Yuxuan Wang,Zhangyu Xiao,Zhijie Yan,Yexin Yang,Bin Zhang,Qinglin Zhang,Shiliang Zhang,Nan Zhao,Siqi Zheng

2024-07-11

Abstract:This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at <a class="link-external link-https" href="https://fun-audio-llm.github.io" rel="external noopener nofollow">this https URL</a>, and the code can be accessed at <a class="link-external link-https" href="https://github.com/FunAudioLLM" rel="external noopener nofollow">this https URL</a>.

Sound,Artificial Intelligence,Audio and Speech Processing

What problem does this paper attempt to address?

The paper primarily aims to address the issue of natural speech interaction between humans and large language models (LLMs). Specifically, the paper proposes the FunAudioLLM framework, which includes two core models:

1. **SenseVoice**: This is an advanced speech understanding model capable of handling tasks such as multilingual speech recognition, emotion recognition, and audio event detection. SenseVoice has two variants:
- **SenseVoice-Small**: Supports multilingual recognition in Chinese, English, Cantonese, Japanese, and Korean, with extremely low latency (less than 80 milliseconds), over 5 times faster than Whisper-small, and over 15 times faster than Whisper-large.
- **SenseVoice-Large**: Supports high-precision speech recognition in over 50 languages, with particularly outstanding performance in Chinese and Cantonese recognition.

2. **CosyVoice**: This is a model for generating natural speech, capable of controlling multiple languages, timbres, speaking styles, and speaker identities. CosyVoice also has the ability to adapt to new speakers with zero samples, cross-language voice cloning capability, and fine-tuning through instruction text. CosyVoice supports five languages: Chinese, English, Japanese, Cantonese, and Korean.

By integrating these models with LLMs, FunAudioLLM can achieve various applications such as speech-to-speech translation, emotional voice chatting, interactive podcasts, and expressive audiobook reading, thereby advancing speech interaction technology. Additionally, the related models have been open-sourced, providing training, inference, and fine-tuning code.