Abstract:This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. Recently, conditional flow matching (CFM) generative models have been successfully adopted for waveform generation tasks, leveraging a single vector field estimation objective for training. Although these models can generate high-fidelity waveform signals, they require significantly more ODE steps compared to GAN-based models, which only need a single generation step. Additionally, the generated samples often lack high-frequency information due to noisy vector field estimation, which fails to ensure high-frequency reproduction. To address this limitation, we enhance pre-trained CFM-based generative models by incorporating a fixed-step generator modification. We utilized reconstruction losses and adversarial feedback to accelerate high-fidelity waveform generation. Through adversarial flow matching optimization, it only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. Moreover, we significantly reduce inference speed from 16 steps to 2 or 4 steps. Additionally, by scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance, with a perceptual evaluation of speech quality (PESQ) score of 4.454 on the LibriTTS dataset. Audio samples, source code and checkpoints will be available at <a class="link-external link-https" href="https://github.com/sh-lee-prml/PeriodWave" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper primarily aims to address the following issues: 1. **Improving waveform generation efficiency**: Although existing Conditional Flow Matching (CFM) models can generate high-quality waveform signals, they require more Ordinary Differential Equation (ODE) steps to generate waveforms compared to Generative Adversarial Networks (GANs) models, resulting in a slower generation process. 2. **Enhancing the retention of high-frequency information**: CFM-based models often lose high-frequency information during the generation process because the noise in vector field estimation affects the retention of high-frequency components. To solve the above problems, the researchers proposed the PeriodWave-Turbo model, an efficient and high-fidelity waveform generation model that combines CFM and GAN techniques. Specifically, the model is improved through the following methods: - **Fixed-step generator modification**: By converting the pre-trained CFM model into a fixed-step generator, the number of steps required to generate waveforms is reduced, thereby speeding up the generation process. - **Reconstruction loss and adversarial feedback**: Utilizing reconstruction loss (such as Mel-spectrogram reconstruction loss) and adversarial training (employing multi-period discriminators and multi-scale sub-band constant-Q transform discriminators) to optimize generation quality, especially in terms of retaining high-frequency information. - **Parameter adjustment and optimization**: Through a small number of fine-tuning steps (e.g., 1000 steps) combined with appropriate hyperparameter settings, the model performance can be significantly improved while greatly reducing training time. Through these methods, PeriodWave-Turbo not only increases the generation speed but also improves the quality of the generated waveforms, particularly in the performance of high-frequency components. Additionally, the scalability of the model demonstrates its robustness and effectiveness across different scales. Ultimately, the model performs excellently on multiple objective and subjective evaluation metrics, including PESQ scores, and shows high efficiency and quality in practical applications.

Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization

PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation

FloWaveNet : A Generative Flow for Raw Audio

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

Waveform generation for text-to-speech synthesis using pitch-synchronous multi-scale generative adversarial networks

WG-WaveNet: Real-Time High-Fidelity Speech Synthesis Without GPU

WaveFlow: A Compact Flow-based Model for Raw Audio

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction

SiD-WaveFlow: A Low-Resource Vocoder Independent of Prior Knowledge

DFlow: A Generative Model Combining Denoising AutoEncoder and Normalizing Flow for High Fidelity Waveform Generation

Probability density distillation with generative adversarial networks for high-quality parallel waveform generation

HiFi-WaveGAN: Generative Adversarial Network with Auxiliary Spectrogram-Phase Loss for High-Fidelity Singing Voice Generation

FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner

WaveCycleGAN: Synthetic-to-natural speech waveform conversion using cycle-consistent adversarial networks

Chunked Autoregressive GAN for Conditional Waveform Synthesis

Efficient WaveGlow: an Improved WaveGlow Vocoder with Enhanced Speed

High Fidelity Speech Synthesis with Adversarial Networks

Parallel Synthesis for Autoregressive Speech Generation

Application Research of Short-Time Fourier Transform in Music Generation Based on the Parallel WaveGan System

A Compact Transformer-Based GAN Vocoder