Abstract:In recent years, DeepFake technology has achieved unprecedented success in high-quality video synthesis, but these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present a formidable challenge to existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often drastically decrease or even fail when tackling lip-syncing videos. In this paper, for the first time, we propose a novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions to boost accuracy. To better illustrate the effectiveness and advances of our proposed method, we create a high-quality LipSync dataset, AVLips, by employing the state-of-the-art lip generators. We hope this high-quality and diverse dataset could be well served the further research on this challenging and interesting field. Experimental results show that our approach gives an average accuracy of more than 95.3% in spotting lip-syncing videos, significantly outperforming the baselines. Extensive experiments demonstrate the capability to tackle deepfakes and the robustness in surviving diverse input transformations. Our method achieves an accuracy of up to 90.2% in real-world scenarios (e.g., WeChat video call) and shows its powerful capabilities in real scenario deployment. To facilitate the progress of this research community, we release all resources at <a class="link-external link-https" href="https://github.com/AaronComo/LipFD" rel="external noopener nofollow">this https URL</a>.

Real-time Lip Synchronization Based on Hidden Markov Models

Real-time speech-driven lip synchronization

VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild

MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting

Audio-driven Talking Face Video Generation with Natural Head Pose

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model

VividWav2Lip: High-Fidelity Facial Animation Generation Based on Speech-Driven Lip Synchronization

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

A Novel Speech-Driven Lip-Sync Model with CNN and LSTM

Real-Time Lip Sync for Live 2D Animation

ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer

SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory

Multimodal Learning for Temporally Coherent Talking Face Generation with Articulator Synergy

Towards Realistic Visual Dubbing with Heterogeneous Sources

Video-audio Driven Real-Time Facial Animation.

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

Style-Preserving Lip Sync via Audio-Aware Style Reference

Lip syncing method for realistic expressive 3D face model

Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes

VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices