Abstract:In this paper, we propose DiffSwap, a diffusion model based framework for high-fidelity and controllable face swapping. Unlike previous work that relies on carefully designed network architectures and loss functions to fuse the information from the source and target faces, we reformulate the face swapping as a conditional inpainting task, performed by a powerful diffusion model guided by the desired face attributes (e.g., identity and landmarks). An important issue that makes it nontrivial to apply diffusion models to face swapping is that we cannot perform the time-consuming multi-step sampling to obtain the generated image during training. To overcome this, we propose a midpoint estimation method to efficiently recover a reasonable diffusion result of the swapped face with only 2 steps, which enables us to introduce identity constraints to improve the face swapping quality. Our framework enjoys several favorable properties more appealing than prior arts: 1) Controllable. Our method is based on conditional masked diffusion on the latent space, where the mask and the conditions can be fully controlled and customized. 2) High-fidelity. The formulation of conditional inpainting can fully exploit the generative ability of diffusion models and can preserve the background of target images with minimal artifacts. 3) Shape-preserving. The controllability of our method enables us to use 3D-aware landmarks as the condition during generation to preserve the shape of the source face. Extensive experiments on both FF++ and FFHQ demonstrate that our method can achieve state-of-the-art face swapping results both qualitatively and quantitatively.

Attribute-Aware Head Swapping Guided by 3d Modeling

FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Designing One Unified Framework for High-Fidelity Face Reenactment and Swapping

Region-Aware Face Swapping

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

3D-Aware Face Swapping

Face Swapping under Large Pose Variations: A 3D Model Based Approach

An Efficient Attribute-Preserving Framework for Face Swapping

FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping

Face swapping based on 3-D photo realistic head reconstruction

FaceController: Controllable Attribute Editing for Face in the Wild

Few-Shot Head Swapping in the Wild

Face Swapping with Adaptive Exploration-Fusion Mechanism and Dual En-Decoing Tactic

DiffSwap: High-Fidelity and Controllable Face Swapping Via 3D-Aware Masked Diffusion

AP-GAN: Improving Attribute Preservation in Video Face Swapping

Face Replacement with Large-Pose Differences.

VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping

Temporal Optimization for Face Swapping Video Based on Consistency Inheritance

Face Swapping: Realistic Image Synthesis Based on Facial Landmarks Alignment

Pose-Free Face Swapping Based On A Deformable 3d Shape Morphable Model

Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models