Abstract:Advanced diffusion-based Text-to-Image (T2I) models, such as the Stable Diffusion Model, have made significant progress in generating diverse and high-quality images using text prompts alone. However, when non-famous users require personalized image generation for their identities (IDs), the T2I models fail to accurately generate their ID-related images. The main problem is that pre-trained T2I models do not learn the mapping between the new ID prompts and their corresponding visual content. The previous methods either failed to accurately fit the face region or lost the interactive generative ability with other existing concepts in T2I models. In other words, they are unable to generate T2I-aligned and semantic-fidelity images for the given prompts with other concepts such as scenes (``Eiffel Tower''), actions (``holding a basketball''), and facial attributes (``eyes closed''). In this paper, we focus on inserting accurate and interactive ID embedding into the Stable Diffusion Model for semantic-fidelity personalized generation. We address this challenge from two perspectives: face-wise region fitting and semantic-fidelity token optimization. Specifically, we first visualize the attention overfit problem and propose a face-wise attention loss to fit the face region instead of entangling ID-unrelated information, such as face layout and background. This key trick significantly enhances the ID accuracy and interactive generative ability with other existing concepts. Then, we optimize one ID representation as multiple per-stage tokens where each token contains two disentangled features. This expansion of the textual conditioning space improves semantic-fidelity control. Extensive experiments validate that our results exhibit superior ID accuracy, text-based manipulation ability, and generalization compared to previous methods.

An Improved Method for Personalizing Diffusion Models

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Low-Rank Continual Personalization of Diffusion Models

Beyond Inserting: Learning Identity Embedding for Semantic-Fidelity Personalized Diffusion Generation

Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations

MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models

ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance

Inserting Anybody in Diffusion Models via Celeb Basis

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models

HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models

FABRIC: Personalizing Diffusion Models with Iterative Feedback

MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration

Fast Personalized Text-to-Image Syntheses With Attention Injection

DreamDistribution: Prompt Distribution Learning for Text-to-Image Diffusion Models

Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

Fast Personalized Text to Image Synthesis with Attention Injection

RoomDiffusion: A Specialized Diffusion Model in the Interior Design Industry

Diffusion Cocktail: Fused Generation from Diffusion Models