Abstract:The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery techniques, varied facial features and complex environmental factors pose significant challenges for face forgery analysis. Existing datasets lack descriptions of these aspects, making it difficult for models to distinguish between real and forged faces using only visual information amid various confounding factors. In addition, existing methods do not yield user-friendly and explainable results, complicating the understanding of the model's decision-making process. To address these challenges, we introduce a novel Open-World Face Forgery Analysis VQA (OW-FFA-VQA) task and the corresponding benchmark. To tackle this task, we first establish a dataset featuring a diverse collection of real and forged face images with essential descriptions and reliable forgery reasoning. Base on this dataset, we introduce FFAA: Face Forgery Analysis Assistant, consisting of a fine-tuned Multimodal Large Language Model (MLLM) and Multi-answer Intelligent Decision System (MIDS). By integrating hypothetical prompts with MIDS, the impact of fuzzy classification boundaries is effectively mitigated, enhancing the model's robustness. Extensive experiments demonstrate that our method not only provides user-friendly explainable results but also significantly boosts accuracy and robustness compared to previous methods.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve include: 1. **Diversity and Uncertainty of Deepfake Technologies**: With the rapid development of deepfake technology, a variety of unknown and diverse forgery techniques have emerged, which makes existing forgery detection models difficult to deal with forged images in the open world. 2. **Complexity of Facial Features and Environmental Factors**: Different human face features (such as makeup, accessories, expressions, and orientations) and complex environmental factors (such as lighting conditions) increase the difficulty of analyzing forged human faces. Existing methods perform poorly under these complex conditions and lack anti - interference ability. 3. **Insufficient Dataset Description**: Existing forged human face datasets lack detailed descriptions of forgery techniques and image features, resulting in models being easily affected by confounding factors when relying solely on visual information for authenticity judgment. 4. **Poor Result Interpretability**: Existing forgery detection methods usually simplify the problem into a binary classification task, with the output result being only true or false, or providing a heat map, which makes it difficult to understand the decision - making process of the model and lacks user - friendliness and interpretability. To address the above challenges, the author introduced a new open - world forged human face analysis task (OW - FFA - VQA) and established a corresponding benchmark dataset (OW - FFA - Bench). Specifically: - **New Task and Dataset**: The author proposed a task combined with Visual Question Answering (VQA), aiming to provide more user - friendly and interpretable results through Multi - Modal Large Language Models (MLLM) and Multi - Answer Intelligent Decision Systems (MIDS). - **Model Improvement**: The author enhanced the robustness of the model by introducing hypothetical prompts and selected the answer that best conforms to the authenticity of the image through MIDS, thereby alleviating the impact of fuzzy classification boundaries. In summary, this paper aims to improve the accuracy and robustness of forged human face analysis while providing more user - friendly and interpretable results by constructing new tasks and datasets and proposing an improved model architecture.

FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant