DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection

Zhiyuan Yan,Yong Zhang,Xinhang Yuan,Siwei Lyu,Baoyuan Wu

2023-10-28

Abstract:A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. This issue leads to unfair performance comparisons and potentially misleading results. Specifically, there is a lack of uniformity in data processing pipelines, resulting in inconsistent data inputs for detection models. Additionally, there are noticeable differences in experimental settings, and evaluation strategies and metrics lack standardization. To fill this gap, we present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions: 1) a unified data management system to ensure consistent input across all detectors, 2) an integrated framework for state-of-the-art methods implementation, and 3) standardized evaluation metrics and protocols to promote transparency and reproducibility. Featuring an extensible, modular-based codebase, DeepfakeBench contains 15 state-of-the-art detection methods, 9 deepfake datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations. Moreover, we provide new insights based on extensive analysis of these evaluations from various perspectives (e.g., data augmentations, backbones). We hope that our efforts could facilitate future research and foster innovation in this increasingly critical domain. All codes, evaluations, and analyses of our benchmark are publicly available at <a class="link-external link-https" href="https://github.com/SCLBD/DeepfakeBench" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper focuses on the challenges in the field of Deepfake detection, specifically the lack of a unified and comprehensive benchmarking framework. The current issues include inconsistent data processing, variations in experimental settings, and non-standardized evaluation criteria and metrics. To address these problems, researchers propose DeepfakeBench, a comprehensive benchmarking platform that provides the following three key contributions:

1. A unified data management system to ensure consistency in inputs from all detectors.
2. An integrated framework to implement state-of-the-art detection methods, enabling direct comparisons between different approaches.
3. Standardized evaluation metrics and protocols to promote transparency and repeatability.

DeepfakeBench includes 15 state-of-the-art detection methods, 9 deepfake datasets, a range of evaluation protocols and analysis tools, as well as comprehensive evaluations. Furthermore, by conducting multidimensional analyses, it offers new insights to drive future research and innovation in this critical domain.

The goal of the paper is to establish a fair platform for comparing deepfake detection techniques, aiming to enhance the reproducibility and accuracy of research while uncovering various factors affecting detection performance, thus facilitating technological advancements and preventing potential misuse.