MagicID: Flexible ID Fidelity Generation System

Zhaoli Deng,Wen Liu,Fanyi Wang,Junkang Zhang,Fan Chen,Meng Zhang,Wendong Zhang,Zhenpeng Mi

2024-08-20

Abstract:Portrait Fidelity Generation is a prominent research area in generative models, with a primary focus on enhancing both controllability and fidelity. Current methods face challenges in generating high-fidelity portrait results when faces occupy a small portion of the image with a low resolution, especially in multi-person group photo settings. To tackle these issues, we propose a systematic solution called MagicID, based on a self-constructed million-level multi-modal dataset named IDZoom. MagicID consists of Multi-Mode Fusion training strategy (MMF) and DDIM Inversion based ID Restoration inference framework (DIIR). During training, MMF iteratively uses the skeleton and landmark modalities from IDZoom as conditional guidance. By introducing the Clone Face Tuning in training stage and Mask Guided Multi-ID Cross Attention (MGMICA) in inference stage, explicit constraints on face positional features are achieved for multi-ID group photo generation. The DIIR aims to address the issue of artifacts. The DDIM Inversion is used in conjunction with face landmarks, global and local face features to achieve face restoration while keeping the background unchanged. Additionally, DIIR is plug-and-play and can be applied to any diffusion-based portrait generation method. To validate the effectiveness of MagicID, we conducted extensive comparative and ablation experiments. The experimental results demonstrate that MagicID has significant advantages in both subjective and objective metrics, and achieves controllable generation in multi-person scenarios.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve two main problems:

1. **Multi - ID position - specified generation**: Current methods have difficulty precisely placing different people in specified positions when generating group photos. Although some methods can achieve position - specified generation through position masks, this may affect the generation quality and still cannot guarantee the accuracy and clarity of identities.

2. **Low - resolution facial detail generation**: When a human face occupies a small proportion or has a low resolution in an image, existing methods are prone to produce obvious artifacts, which greatly limits the practical applications of these methods. This problem is more prominent especially in multi - person scenes.

To solve these problems, the author proposes a new flexible ID - fidelity generation system, named **MagicID**. This system combines the **Multi - Modal Fusion training strategy (MMF)** and the **DDIM - inversion - based ID - inpainting Inference Framework (DIIR)**. Specifically:

- **Multi - Modal Fusion training strategy (MMF)**: By introducing Clone Face Tuning and Mask - Guided Multi - ID Cross - Attention (MGMICA), explicit constraints on facial position features are achieved during the training and inference stages respectively. This enables the model to generate high - quality images in multi - person scenes and ensure that each ID is located in the specified position.

- **DDIM - inversion - based ID - inpainting Inference Framework (DIIR)**: This framework aims to repair artifacts in low - resolution faces while keeping the background unchanged. By combining facial key points, global and local facial features, DIIR can restore facial details without damaging the background information, thereby significantly improving the quality of the generated images.

In addition, the author also constructs a data engine **IDZoom** containing millions of samples. This dataset contains six - modal data: titles, human body images, skeletons, face images, facial key points, and face area masks. These innovations make MagicID perform excellently in generation tasks in both single - person and multi - person scenes and achieve significant advantages in multiple evaluation metrics.

### Summary
The main contributions of MagicID are:
- Proposing the Multi - Modal Fusion training strategy (MMF) and Mask - Guided Multi - ID Cross - Attention (MGMICA), which solves the problem of multi - ID position - specified generation.
- Introducing the DDIM - inversion - based ID - inpainting Inference Framework (DIIR), which effectively solves the problem of artifacts in low - resolution faces.
- Constructing a large - scale multi - modal dataset IDZoom, which provides rich data support for model training.

Through these innovations, MagicID performs excellently in both single - person bust and multi - person photo - generation tasks, especially making significant progress in low - resolution face inpainting.