Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio

Yi Lu,Yuankun Xie,Ruibo Fu,Zhengqi Wen,Jianhua Tao,Zhiyong Wang,Xin Qi,Xuefei Liu,Yongwei Li,Yukun Liu,Xiaopeng Wang,Shuchen Shi

2024-06-12

Abstract:With the proliferation of Large Language Model (LLM) based deepfake audio, there is an urgent need for effective detection methods. Previous deepfake audio generation methods typically involve a multi-step generation process, with the final step using a vocoder to predict the waveform from handcrafted features. However, LLM-based audio is directly generated from discrete neural codecs in an end-to-end generation process, skipping the final step of vocoder processing. This poses a significant challenge for current audio deepfake detection (ADD) models based on vocoder artifacts. To effectively detect LLM-based deepfake audio, we focus on the core of the generation process, the conversion from neural codec to waveform. We propose Codecfake dataset, which is generated by seven representative neural codec methods. Experiment results show that codec-trained ADD models exhibit a 41.406% reduction in average equal error rate compared to vocoder-trained ADD models on the Codecfake test set.

Sound,Artificial Intelligence,Audio and Speech Processing

What problem does this paper attempt to address?

The problem this paper attempts to address is the performance degradation of current Audio Deepfake Detection (ADD) models when faced with audio forgeries generated by large-scale language models (LLMs). Traditional audio forgery generation methods typically involve multiple steps, ultimately predicting waveforms from handcrafted features through a vocoder. However, LLM-based audio generation methods directly produce audio from a discrete neural codec, bypassing the final vocoder processing step. This results in existing ADD models, which rely on vocoder artifacts, performing poorly in detecting LLM-generated audio forgeries.

To effectively detect LLM-based audio forgeries, the authors focus on the core part of the generation process, which is the conversion from neural codec to waveform. They propose a dataset named Codecfake, which is generated by seven representative neural codec methods. Experimental results show that ADD models trained with codecs achieve an average Equal Error Rate (EER) on the Codecfake test set that is 41.406% lower than that of ADD models trained with vocoders.