HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec

Dongchao Yang,Songxiang Liu,Rongjie Huang,Jinchuan Tian,Chao Weng,Yuexian Zou

2023-05-07

Abstract:Audio codec models are widely used in audio communication as a crucial technique for compressing audio into discrete representations. Nowadays, audio codec models are increasingly utilized in generation fields as intermediate representations. For instance, AudioLM is an audio generation model that uses the discrete representation of SoundStream as a training target, while VALL-E employs the Encodec model as an intermediate feature to aid TTS tasks. Despite their usefulness, two challenges persist: (1) training these audio codec models can be difficult due to the lack of publicly available training processes and the need for large-scale data and GPUs; (2) achieving good reconstruction performance requires many codebooks, which increases the burden on generation models. In this study, we propose a group-residual vector quantization (GRVQ) technique and use it to develop a novel \textbf{Hi}gh \textbf{Fi}delity Audio Codec model, HiFi-Codec, which only requires 4 codebooks. We train all the models using publicly available TTS data such as LibriTTS, VCTK, AISHELL, and more, with a total duration of over 1000 hours, using 8 GPUs. Our experimental results show that HiFi-Codec outperforms Encodec in terms of reconstruction performance despite requiring only 4 codebooks. To facilitate research in audio codec and generation, we introduce AcademiCodec, the first open-source audio codec toolkit that offers training codes and pre-trained models for Encodec, SoundStream, and HiFi-Codec. Code and pre-trained model can be found on: \href{<a class="link-external link-https" href="https://github.com/yangdongchao/AcademiCodec" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/yangdongchao/AcademiCodec" rel="external noopener nofollow">this https URL</a>}

Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The paper primarily aims to address two core challenges faced by audio codecs in generative tasks:

1. **Training Difficulty**: Existing audio codec models are difficult to train due to reasons such as the lack of publicly available training procedures, the need for large-scale datasets, and the requirement for high-performance GPUs.
2. **Number of Codebooks**: To achieve good reconstruction performance, many existing audio codecs need to use a large number of codebooks, which burdens the generative model.

To address the above issues, the researchers proposed a method called **Group-Residual Vector Quantization (GRVQ)** and developed a new high-fidelity audio codec model based on this method—**HiFi-Codec**. This model aims to simplify the audio reconstruction process in generative tasks by reducing the number of required codebooks while maintaining or improving reconstruction quality.

Specifically, the innovation of HiFi-Codec lies in its GRVQ technology, which first divides the latent features into multiple groups and then applies residual vector quantization (RVQ) to each group separately. This approach allows the codebooks in the first layer to store more information, thereby reducing the number of codebooks needed in subsequent layers. Experimental results show that the HiFi-Codec model can achieve or even surpass the reconstruction performance level of existing models (such as Encodec) using only 4 codebooks.

Additionally, the research team has released **AcademicoCodec**, the first open-source audio codec toolkit, which includes the training code and pre-trained models for Encodec, SoundStream, and HiFi-Codec, to promote research development in the related field.