IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities

Xin Zhang,Xiang Lyu,Zhihao Du,Qian Chen,Dong Zhang,Hangrui Hu,Chaohong Tan,Tianyu Zhao,Yuxuan Wang,Bin Zhang,Heng Lu,Yaqian Zhou,Xipeng Qiu

2024-10-12

Abstract:Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain content quality, which unfortunately brings computational overhead and increases latency in multi-turn interactions. To address this, we introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. IntrinsicVoice aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences while generating high-quality audio, significantly reducing the length difference between speech and text, speeding up inference, and alleviating long-text modeling issues. Additionally, we construct a multi-turn speech-to-speech dialogue dataset named \method-500k which includes nearly 500k turns of speech-to-speech dialogues, and a cross-modality training strategy to enhance the semantic alignment between speech and text. Experimental results demonstrate that IntrinsicVoice can generate high-quality speech response with latency lower than 100ms in multi-turn dialogue scenarios. Demos are available at <a class="link-external link-https" href="https://instrinsicvoice.github.io/" rel="external noopener nofollow">this https URL</a>.

Sound,Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is: how to endow large language models (LLMs) with low-latency, high-quality real-time voice interaction capabilities, especially in multi-turn dialogue scenarios. Current methods typically rely on explicit text autoregressive generation to maintain content quality, which not only increases computational overhead but also leads to increased latency in multi-turn interactions. To overcome these issues, the authors propose IntrinsicVoice, a large language model with inherent real-time voice interaction capabilities.

Specifically, IntrinsicVoice aims to achieve efficient, low-latency voice interaction by reducing the modality gap between voice sequences and text sequences, thereby transferring the text capabilities of pre-trained LLMs to the voice modality. The authors introduce a novel architecture—GroupFormer, which can compress voice sequences to a length comparable to text sequences while generating high-quality audio. This significantly reduces the length disparity between voice and text, speeds up inference, and alleviates the problem of long text modeling. Additionally, they constructed a dataset containing nearly 500,000 rounds of voice dialogues, IntrinsicVoice-500k, and proposed a cross-modal training strategy to enhance semantic alignment between voice and text.

Experimental results show that IntrinsicVoice can generate high-quality voice responses in multi-turn dialogue scenarios with latency below 100 milliseconds, achieving true real-time voice interaction.