A Sanity Check for AI-generated Image Detection

Shilin Yan,Ouxiang Li,Jiayin Cai,Yanbin Hao,Xiaolong Jiang,Yao Hu,Weidi Xie

2024-10-08

Abstract:With the rapid development of generative models, discerning AI-generated content has evoked increasing attention from both industry and academia. In this paper, we conduct a sanity check on "whether the task of AI-generated image detection has been solved". To start with, we present Chameleon dataset, consisting AIgenerated images that are genuinely challenging for human perception. To quantify the generalization of existing methods, we evaluate 9 off-the-shelf AI-generated image detectors on Chameleon dataset. Upon analysis, almost all models classify AI-generated images as real ones. Later, we propose AIDE (AI-generated Image DEtector with Hybrid Features), which leverages multiple experts to simultaneously extract visual artifacts and noise patterns. Specifically, to capture the high-level semantics, we utilize CLIP to compute the visual embedding. This effectively enables the model to discern AI-generated images based on semantics or contextual information; Secondly, we select the highest frequency patches and the lowest frequency patches in the image, and compute the low-level patchwise features, aiming to detect AI-generated images by low-level artifacts, for example, noise pattern, anti-aliasing, etc. While evaluating on existing benchmarks, for example, AIGCDetectBenchmark and GenImage, AIDE achieves +3.5% and +4.6% improvements to state-of-the-art methods, and on our proposed challenging Chameleon benchmarks, it also achieves the promising results, despite this problem for detecting AI-generated images is far from being solved. The dataset, codes, and pre-train models will be published at <a class="link-external link-https" href="https://github.com/shilinyan99/AIDE" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper primarily focuses on how to distinguish between AI-generated images and real-world images. Specifically:

1. **Proposing a Challenging Dataset**:
- **Chameleon Dataset**: Contains a series of AI-generated images that are highly deceptive to human perception. These images are carefully selected and have passed the "Turing Test," meaning human annotators find it difficult to distinguish their authenticity.
- The goal is to evaluate the performance of existing methods when faced with highly realistic AI-generated images.

2. **Redefining Task Settings**:
- Existing methods are usually trained on specific types of generative models (such as GANs or diffusion models), whereas in real-world applications, it is necessary to detect images from various generative models.
- A new training-testing setup is proposed, allowing models to be trained on multiple generative models and tested on the Chameleon dataset to better simulate real-world challenges.

3. **Proposing a New Method AIDE**:
- **AIDE (AI-generated Image Detector with Hybrid Features)**: Combines low-level texture features and high-level semantic features to detect AI-generated images.
- Through experimental validation, AIDE outperforms current state-of-the-art methods on existing benchmark datasets (such as AIGCDetectBenchmark and GenImage) and also performs excellently on the Chameleon dataset.

Overall, the paper aims to reveal the challenges present in the current AI-generated image detection tasks and proposes a more robust method to address these challenges.