Abstract:In this paper, we propose a deep learning based system for the task of deepfake audio detection. In particular, the draw input audio is first transformed into various spectrograms using three transformation methods of Short-time Fourier Transform (STFT), Constant-Q Transform (CQT), Wavelet Transform (WT) combined with different auditory-based filters of Mel, Gammatone, linear filters (LF), and discrete cosine transform (DCT). Given the spectrograms, we evaluate a wide range of classification models based on three deep learning approaches. The first approach is to train directly the spectrograms using our proposed baseline models of CNN-based model (CNN-baseline), RNN-based model (RNN-baseline), C-RNN model (C-RNN baseline). Meanwhile, the second approach is transfer learning from computer vision models such as ResNet-18, MobileNet-V3, EfficientNet-B0, DenseNet-121, SuffleNet-V2, Swint, Convnext-Tiny, GoogLeNet, MNASsnet, RegNet. In the third approach, we leverage the state-of-the-art audio pre-trained models of Whisper, Seamless, Speechbrain, and Pyannote to extract audio embeddings from the input spectrograms. Then, the audio embeddings are explored by a Multilayer perceptron (MLP) model to detect the fake or real audio samples. Finally, high-performance deep learning models from these approaches are fused to achieve the best performance. We evaluated our proposed models on ASVspoof 2019 benchmark dataset. Our best ensemble model achieved an Equal Error Rate (EER) of 0.03, which is highly competitive to top-performing systems in the ASVspoofing 2019 challenge. Experimental results also highlight the potential of selective spectrograms and deep learning approaches to enhance the task of audio deepfake detection.

Porn Streamer Audio Recognition Based on Deep Learning and Random Forest

Multilevel fusion of multimodal deep features for porn streamer recognition in live video

Sensitive Information Detection Based on Deep Learning Models

What Did I Just Hear? Detecting Pornographic Sounds in Adult Videos Using Neural Networks

Fusing Audio-Words with Visual Features for Pornographic Video Detection

Transferring Audio Deepfake Detection Capability Across Languages

Streaming Active Deep Forest for Evolving Data Stream Classification

Learning Deep Embedding with Acoustic and Phoneme Features for Speaker Recognition in FM Broadcasting

LCMA-Net: A light cross-modal attention network for streamer re-identification in live video

Speaker Recognition-Assisted Robust Audio Deepfake Detection

Generalized Source Tracing: Detecting Novel Audio Deepfake Algorithm with Real Emphasis and Fake Dispersion Strategy

Simulation of English Speech Recognition Based on Improved Extreme Random Forest Classification

Applying deep learning to classify pornographic images and videos

Deepfake Audio Detection Using Spectrogram-based Feature and Ensemble of Deep Learning Models

Multi-Scale Permutation Entropy for Audio Deepfake Detection

Popularity, face and voice: Predicting and interpreting livestreamers' retail performance using machine learning techniques

Combining Information from Multi-Stream Features Using Deep Neural Network in Speech Recognition

Multi-Modal Machine Learning for Assessing Gaming Skills in Online Streaming: A Case Study with CS:GO

Pornprobe: an LDA-SVM based pornography detection system.

Auditory Context Classification Using Random Forests

A Three-Stream CNN-LSTM Network for Automatic Modulation Classification