Abstract:Aim: The purpose of the article is to present the hypothesis that the use of discrepancies in audiovisual materials can significantly increase the effectiveness of detecting various types of deepfake and related threats. In order to verify this hypothesis, the authors proposed a new method that reveals inconsistencies in both multiple modalities simultaneously and within individual modalities separately, enabling them to effectively distinguish between authentic and altered public speaking videos. Project and methods: The proposed approach is to integrate audio and visual signals in a so-called fine-grained manner, and then carry out binary classification processes based on calculated adjustments to the classification results of each modality. The method has been tested using various network architectures, in particular Capsule networks – for deep anomaly detection and Swin Transformer – for image classification. Pre-processing included frame extraction and face detection using the MTCNN algorithm, as well as conversion of audio to mel spectrograms to better reflect human auditory perception. The proposed technique was tested on multimodal deepfake datasets, namely FakeAVCeleb and TMC, along with a custom dataset containing 4,700 recordings. The method has shown high performance in identifying deepfake threats in various test scenarios. Results: The method proposed by the authors achieved better AUC and accuracy compared to other reference methods, confirming its effectiveness in the analysis of multimodal artefacts. The test results confirm that it is effective in detecting modified videos in a variety of test scenarios which can be considered an advance over existing deepfake detection techniques. The results highlight the adaptability of the method in various architectures of feature extraction networks. Conclusions: The presented method of audiovisual deepfake detection uses fine inconsistencies of multimodal features to distinguish whether the material is authentic or synthetic. It is distinguished by its ability to point out inconsistencies in different types of deepfakes and, within each individual modality, can effectively distinguish authentic content from manipulated counterparts. The adaptability has been confirmed by the successful application of the method in various feature extraction network architectures. Moreover, its effectiveness has been proven in rigorous tests on two different audiovisual deepfake datasets. Keywords: analysis of audio-video stream, detection of deepfake threats, analysis of public speeches

Detecting Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data

A Context-Aware Approach for Detecting Check-Worthy Claims in Political Debates

Towards Effective Rebuttal: Listening Comprehension using Corpus-Wide Claim Mining

A Multimodal Predictive Model of Successful Debaters or How I Learned to Sway Votes

A Recorded Debating Dataset

Audio-Video Analysis Method of Public Speaking Videos to Detect Deepfake Threat

Automatic Fact-Checking Using Context and Discourse Information

Out of the Echo Chamber: Detecting Countering Debate Speeches

Sound Check: Auditing Audio Datasets

Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Video

Testing the Validity of Automatic Speech Recognition for Political Text Analysis

Television Discourse Decoded: Comprehensive Multimodal Analytics at Scale

Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics

Predicting the Leading Political Ideology of YouTube Channels Using Acoustic, Textual, and Metadata Information

Multilingual and Multimodal Abuse Detection

Visual and audio scene classification for detecting discrepancies in video: a baseline method and experimental protocol

Multilingual and Multi-topical Benchmark of Fine-tuned Language models and Large Language Models for Check-Worthy Claim Detection

Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text

Audio Retrieval with Natural Language Queries: A Benchmark Study

Hear Me Out: A Study on the Use of the Voice Modality for Crowdsourced Relevance Assessments

Checkworthiness in Automatic Claim Detection Models: Definitions and Analysis of Datasets