Abstract:In this paper, we propose DiffSwap, a diffusion model based framework for high-fidelity and controllable face swapping. Unlike previous work that relies on carefully designed network architectures and loss functions to fuse the information from the source and target faces, we reformulate the face swapping as a conditional inpainting task, performed by a powerful diffusion model guided by the desired face attributes (e.g., identity and landmarks). An important issue that makes it nontrivial to apply diffusion models to face swapping is that we cannot perform the time-consuming multi-step sampling to obtain the generated image during training. To overcome this, we propose a midpoint estimation method to efficiently recover a reasonable diffusion result of the swapped face with only 2 steps, which enables us to introduce identity constraints to improve the face swapping quality. Our framework enjoys several favorable properties more appealing than prior arts: 1) Controllable. Our method is based on conditional masked diffusion on the latent space, where the mask and the conditions can be fully controlled and customized. 2) High-fidelity. The formulation of conditional inpainting can fully exploit the generative ability of diffusion models and can preserve the background of target images with minimal artifacts. 3) Shape-preserving. The controllability of our method enables us to use 3D-aware landmarks as the condition during generation to preserve the shape of the source face. Extensive experiments on both FF++ and FFHQ demonstrate that our method can achieve state-of-the-art face swapping results both qualitatively and quantitatively.

Face swapping based on 3-D photo realistic head reconstruction

Face Swapping under Large Pose Variations: A 3D Model Based Approach

A synthesis method for personalized 3D face reconstruction

Pose-Free Face Swapping Based On A Deformable 3d Shape Morphable Model

FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Face Replacement with Large-Pose Differences.

Region-Aware Face Swapping

Replacement of Facial Parts in Images.

HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping

Designing One Unified Framework for High-Fidelity Face Reenactment and Swapping

Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models

Attribute-Aware Head Swapping Guided by 3d Modeling

Realistic Reconstruction of Human Face Based on Images

FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping

DiffSwap: High-Fidelity and Controllable Face Swapping Via 3D-Aware Masked Diffusion

Face Transformer: Towards High Fidelity and Accurate Face Swapping

Creating 3d Realistic Head: From Two Orthogonal Photos To Multi-View Face Contents

Few-Shot Head Swapping in the Wild

Automatic Flexible Face Replacement with No Auxiliary Data

High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies

Face Swap via Diffusion Model