LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes

Trung Dang,David Aponte,Dung Tran,Kazuhito Koishida

2024-06-10

Abstract:Prior works have demonstrated zero-shot text-to-speech by using a generative language model on audio tokens obtained via a neural audio codec. It is still challenging, however, to adapt them to low-latency scenarios. In this paper, we present LiveSpeech - a fully autoregressive language model-based approach for zero-shot text-to-speech, enabling low-latency streaming of the output audio. To allow multiple token prediction within a single decoding step, we propose (1) using adaptive codebook loss weights that consider codebook contribution in each frame and focus on hard instances, and (2) grouping codebooks and processing groups in parallel. Experiments show our proposed models achieve competitive results to state-of-the-art baselines in terms of content accuracy, speaker similarity, audio quality, and inference speed while being suitable for low-latency streaming applications.

Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The paper focuses on the challenges of low-latency zero-shot text-to-speech (TTS) technology. Although existing methods can achieve zero-shot TTS, they perform poorly in real-time or low-latency scenarios. The paper proposes a method called LiveSpeech, which is a fully autoregressive language model based on audio discrete coding for low-latency speech output. To predict multiple encodings in a single decoding step, the paper introduces two techniques:

1. Adaptive codebook loss weights: dynamically adjust the model capacity based on the contribution of each codebook and the importance of difficult examples in each frame, allowing the model to efficiently handle up to 16 codebooks in each frame without increasing inference cost.

2. Codebook group parallel processing: enhance the single-step capability by parallel processing codebook groups, which increases the computational workload but allows for parallel prediction of codes within these groups without significantly increasing the inference time.

Experiments show that the LiveSpeech model competes with state-of-the-art baselines in terms of content accuracy, speaker similarity, audio quality, and inference speed, making it suitable for low-latency streaming applications.