Abstract:In this paper, we propose DiffSwap, a diffusion model based framework for high-fidelity and controllable face swapping. Unlike previous work that relies on carefully designed network architectures and loss functions to fuse the information from the source and target faces, we reformulate the face swapping as a conditional inpainting task, performed by a powerful diffusion model guided by the desired face attributes (e.g., identity and landmarks). An important issue that makes it nontrivial to apply diffusion models to face swapping is that we cannot perform the time-consuming multi-step sampling to obtain the generated image during training. To overcome this, we propose a midpoint estimation method to efficiently recover a reasonable diffusion result of the swapped face with only 2 steps, which enables us to introduce identity constraints to improve the face swapping quality. Our framework enjoys several favorable properties more appealing than prior arts: 1) Controllable. Our method is based on conditional masked diffusion on the latent space, where the mask and the conditions can be fully controlled and customized. 2) High-fidelity. The formulation of conditional inpainting can fully exploit the generative ability of diffusion models and can preserve the background of target images with minimal artifacts. 3) Shape-preserving. The controllability of our method enables us to use 3D-aware landmarks as the condition during generation to preserve the shape of the source face. Extensive experiments on both FF++ and FFHQ demonstrate that our method can achieve state-of-the-art face swapping results both qualitatively and quantitatively.

Semantics-guided Generative Diffusion Model with a 3DMM Model Condition for Face Swapping

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Region-Aware Face Swapping

DiffSwap: High-Fidelity and Controllable Face Swapping Via 3D-Aware Masked Diffusion

HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping

HS-Diffusion: Learning a Semantic-Mixing Diffusion Model for Head Swapping

Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models

Learning Disentangled Representation for One-shot Progressive Face Swapping

LDFaceNet: Latent Diffusion-based Network for High-Fidelity Deepfake Generation

Identity-Preserving Face Swapping via Dual Surrogate Generative Models

High-resolution Face Swapping via Latent Semantics Disentanglement

FSNet: An Identity-Aware Generative Model for Image-based Face Swapping

StyleIPSB: Identity-Preserving Semantic Basis of StyleGAN for High Fidelity Face Swapping

Semantic Image Synthesis Via Diffusion Models

Face Swap via Diffusion Model

On Learning 3D Face Morphable Model from In-the-wild Images

MFIM: Megapixel Facial Identity Manipulation

Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control

High-Fidelity Face Swapping with Style Blending

Fine-Grained Face Swapping via Regional GAN Inversion