Abstract:Face inpainting is important in various applications, such as photo restoration, image editing, and virtual reality. Despite the significant advances in face generative models, ensuring that a person's unique facial identity is maintained during the inpainting process is still an elusive goal. Current state-of-the-art techniques, exemplified by MyStyle, necessitate resource-intensive fine-tuning and a substantial number of images for each new identity. Furthermore, existing methods often fall short in accommodating user-specified semantic attributes, such as beard or expression. To improve inpainting results, and reduce the computational complexity during inference, this paper proposes the use of Parallel Visual Attention (PVA) in conjunction with diffusion models. Specifically, we insert parallel attention matrices to each cross-attention module in the denoising network, which attends to features extracted from reference images by an identity encoder. We train the added attention modules and identity encoder on CelebAHQ-IDI, a dataset proposed for identity-preserving face inpainting. Experiments demonstrate that PVA attains unparalleled identity resemblance in both face inpainting and face inpainting with language guidance tasks, in comparison to various benchmarks, including MyStyle, Paint by Example, and Custom Diffusion. Our findings reveal that PVA ensures good identity preservation while offering effective language-controllability. Additionally, in contrast to Custom Diffusion, PVA requires just 40 fine-tuning steps for each new identity, which translates to a significant speed increase of over 20 times.

Fine-Grained Blind Face Inpainting with 3D Face Component Disentanglement

Towards Interactive Facial Image Inpainting by Text or Exemplar Image.

Face Inpainting Via Nested Generative Adversarial Networks

Masked Face Inpainting Through Residual Attention UNet

Face Image Inpainting Based on Generative Adversarial Network

Driving Mutual Advancement of 3D Reconstruction and Inpainting for Masked Faces

Deep Face Video Inpainting via UV Mapping

Reference-Guided Large-Scale Face Inpainting with Identity and Texture Control

ShowFace: Coordinated Face Inpainting with Memory-Disentangled Refinement Networks

FT-TDR: Frequency-guided Transformer and Top-Down Refinement Network for Blind Face Inpainting

Face Frontalization with Inpainting Method

Recovery-Based Occluded Face Recognition by Identity-Guided Inpainting

Face inpainting based on GAN by facial prediction and fusion as guidance information

LaFIn: Generative Landmark Guided Face Inpainting

Face Inpainting With Dynamic Structural Information Of Facial Action Units

Domain Embedded Multi-model Generative Adversarial Networks for Image-based Face Inpainting

Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention

Image Fine-grained Inpainting.

Efficient Generative Face Completion with Perceptual Deblurring

Face Image Inpainting Using Attribute Guided