Abstract:Recent advancements in Latent Diffusion Models (LDMs) have propelled them to the forefront of various generative tasks. However, their iterative sampling process poses a significant computational burden, resulting in slow generation speeds and limiting their application in text-to-audio generation deployment. In this work, we introduce AudioLCM, a novel consistency-based model tailored for efficient and high-quality text-to-audio generation. Unlike prior approaches that address noise removal through iterative processes, AudioLCM integrates Consistency Models (CMs) into the generation process, facilitating rapid inference through a mapping from any point at any time step to the trajectory's initial point. To overcome the convergence issue inherent in LDMs with reduced sample iterations, we propose the Guided Latent Consistency Distillation with a multi-step Ordinary Differential Equation (ODE) solver. This innovation shortens the time schedule from thousands to dozens of steps while maintaining sample quality, thereby achieving fast convergence and high-quality generation. Furthermore, to optimize the performance of transformer-based neural network architectures, we integrate the advanced techniques pioneered by LLaMA into the foundational framework of transformers. This architecture supports stable and efficient training, ensuring robust performance in text-to-audio synthesis. Experimental results on text-to-audio generation and text-to-music synthesis tasks demonstrate that AudioLCM needs only 2 iterations to synthesize high-fidelity audios, while it maintains sample quality competitive with state-of-the-art models using hundreds of steps. AudioLCM enables a sampling speed of 333x faster than real-time on a single NVIDIA 4090Ti GPU, making generative models practically applicable to text-to-audio generation deployment. Our extensive preliminary analysis shows that each design in AudioLCM is effective. https://AudioLCM.github.io/. Code is Available https://github.com/Text-to-Audio/AudioLCM

MusicLM: Generating Music From Text

Simple and Controllable Music Generation

MusicRL: Aligning Music Generation to Human Preferences

AudioLM: a Language Modeling Approach to Audio Generation

Text2midi: Generating Symbolic Music from Captions

MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies

Efficient Neural Music Generation

Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset

Melody-Guided Music Generation

ChatMusician: Understanding and Generating Music Intrinsically with LLM

Intelligent Text-Conditioned Music Generation

MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation

Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models

Audio Conditioning for Music Generation via Discrete Bottleneck Features

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

Accompanied Singing Voice Synthesis with Fully Text-controlled Melody

MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models

AudioLCM: Efficient and High-Quality Text-to-Audio Generation with Minimal Inference Steps

Mustango: Toward Controllable Text-to-Music Generation

InstrumentGen: Generating Sample-Based Musical Instruments From Text

Melody generation based on deep ensemble learning using varying temporal context length