AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

Haohe Liu,Zehua Chen,Yi Yuan,Xinhao Mei,Xubo Liu,Danilo Mandic,Wenwu Wang,Mark D. Plumbley

2023-09-09

Abstract:Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at <a class="link-external link-https" href="https://audioldm.github.io" rel="external noopener nofollow">this https URL</a>.

Sound,Artificial Intelligence,Multimedia,Audio and Speech Processing,Signal Processing

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the quality and computational efficiency of the Text - to - Audio (TTA) generation system. Specifically, the existing TTA systems have the following problems:

1. **Limited generation quality**: Previous TTA systems are not ideal in terms of the quality of generated audio.
2. **High computational cost**: These systems have a relatively high computational complexity, resulting in large resource consumption during the training and inference processes.
3. **Dependence on high - quality audio - text pair data**: Most TTA systems require a large amount of high - quality audio - text pair data for training, but such data is often difficult to obtain and limited in quantity.
4. **Lack of flexibility**: The existing TTA systems have relatively limited functions in audio manipulation (such as style transfer, super - resolution, etc.).

To solve these problems, the paper proposes a new method named AudioLDM. AudioLDM improves the TTA system in the following ways:

- **Using continuous Latent Diffusion Models (LDMs)**: Compared with discrete representations, the continuous latent space can model high - dimensional audio signals more efficiently, thereby improving the generation quality and computational efficiency.
- **Utilizing Contrastive Language - Audio Pretraining (CLAP) embeddings**: Through CLAP embeddings, AudioLDM can use only audio data during the training process without relying on audio - text pair data, thus reducing the demand for large - scale paired data.
- **Achieving zero - shot audio operations**: AudioLDM is able to perform various zero - shot text - guided audio operations, such as style transfer, super - resolution, and inpainting, without the need for fine - tuning for specific tasks.

In summary, AudioLDM aims to significantly improve the generation quality, computational efficiency, and flexibility of the TTA system by introducing new model architectures and techniques.