Abstract:Despite promising progress in face swapping task, realistic swapped images remain elusive, often marred by artifacts, particularly in scenarios involving high pose variation, color differences, and occlusion. To address these issues, we propose a novel approach that better harnesses diffusion models for face-swapping by making following core contributions. (a) We propose to re-frame the face-swapping task as a self-supervised, train-time inpainting problem, enhancing the identity transfer while blending with the target image. (b) We introduce a multi-step Denoising Diffusion Implicit Model (DDIM) sampling during training, reinforcing identity and perceptual similarities. (c) Third, we introduce CLIP feature disentanglement to extract pose, expression, and lighting information from the target image, improving fidelity. (d) Further, we introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping, with an additional feature of head swapping. Ours can swap hair and even accessories, beyond traditional face swapping. Unlike prior works reliant on multiple off-the-shelf models, ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models. Extensive experiments on FFHQ and CelebA datasets validate the efficacy and robustness of our approach, showcasing high-fidelity, realistic face-swapping with minimal inference time. Our code is available at <a class="link-external link-https" href="https://github.com/Sanoojan/REFace" rel="external noopener nofollow">this https URL</a>.

Automatic Flexible Face Replacement with No Auxiliary Data

Replacement of Facial Parts in Images.

Face Replacement with Large-Pose Differences.

FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Automatic 3D Face Modeling Using 2D Active Appearance Models

FReeNet: Multi-Identity Face Reenactment

Region-Aware Face Swapping

Pose-Free Face Swapping Based On A Deformable 3d Shape Morphable Model

Face swapping based on 3-D photo realistic head reconstruction

Designing One Unified Framework for High-Fidelity Face Reenactment and Swapping

An experimental facial synthesis system using graph cut and gradient domain fusion

Realistic Reconstruction of Human Face Based on Images

Automatic Face Modeling and Synthesis Based on Image Pairs

Face Swapping under Large Pose Variations: A 3D Model Based Approach

FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping

Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models

3D human face transplanting via depth camera

Controllable high-fidelity facial performance transfer

HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping

3D-Aware Face Swapping

Automatic 3D Face Modeling from Video