Abstract:Explainability in artificial intelligence is crucial for restoring trust, particularly in areas like face forgery detection, where viewers often struggle to distinguish between real and fabricated content. Vision and Large Language Models (VLLM) bridge computer vision and natural language, offering numerous applications driven by strong common-sense reasoning. Despite their success in various tasks, the potential of vision and language remains underexplored in face forgery detection, where they hold promise for enhancing explainability by leveraging the intrinsic reasoning capabilities of language to analyse fine-grained manipulation areas. As such, there is a need for a methodology that converts face forgery detection to a Visual Question Answering (VQA) task to systematically and fairly evaluate these capabilities. Previous efforts for unified benchmarks in deepfake detection have focused on the simpler binary task, overlooking evaluation protocols for fine-grained detection and text-generative models. We propose a multi-staged approach that diverges from the traditional binary decision paradigm to address this gap. In the first stage, we assess the models' performance on the binary task and their sensitivity to given instructions using several prompts. In the second stage, we delve deeper into fine-grained detection by identifying areas of manipulation in a multiple-choice VQA setting. In the third stage, we convert the fine-grained detection to an open-ended question and compare several matching strategies for the multi-label classification task. Finally, we qualitatively evaluate the fine-grained responses of the VLLMs included in the benchmark. We apply our benchmark to several popular models, providing a detailed comparison of binary, multiple-choice, and open-ended VQA evaluation across seven datasets. \url{<a class="link-external link-https" href="https://nickyfot.github.io/hitchhickersguide.github.io/" rel="external noopener nofollow">this https URL</a>}

Four Principles of Explainable AI as Applied to Biometrics and Facial Forensic Algorithms

Explainable biometrics: a systematic literature review

Visual Psychophysics for Making Face Recognition Algorithms More Explainable

From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing

Bridging Human Concepts and Computer Vision for Explainable Face Verification

Advances in Explainable, Fair, and Trustworthy AI

Towards Design Principles for User-Centric Explainable AI in Fraud Detection

Interpretable algorithmic forensics

Flexible and Context-Specific AI Explainability: A Multidisciplinary Approach

Explainable Artificial Intelligence and Machine Learning: A reality rooted perspective

Explainable AI for Psychological Profiling from Digital Footprints: A Case Study of Big Five Personality Predictions from Spending Data

Explaining Bias in Deep Face Recognition via Image Characteristics

Automated face recognition in forensic science: Review and perspectives

XAI-FR: Explainable AI-Based Face Recognition Using Deep Neural Networks

Principles of Explanation in Human-AI Systems

The ethical application of biometric facial recognition technology

Explainable Artificial Intelligence: Concepts, Applications, Research Challenges and Visions

From understanding to justifying: Computational reliabilism for AI-based forensic evidence evaluation

Towards Fair and Explainable AI using a Human-Centered AI Approach

Explainable AI And Visual Reasoning: Insights From Radiology

A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning