Abstract:Personalized text-to-image generation methods can generate customized images based on the reference images, which have garnered wide research interest. Recent methods propose a finetuning-free approach with a decoupled cross-attention mechanism to generate personalized images requiring no test-time finetuning. However, when multiple reference images are provided, the current decoupled cross-attention mechanism encounters the object confusion problem and fails to map each reference image to its corresponding object, thereby seriously limiting its scope of application. To address the object confusion problem, in this work we investigate the relevance of different positions of the latent image features to the target object in diffusion model, and accordingly propose a weighted-merge method to merge multiple reference image features into the corresponding objects. Next, we integrate this weighted-merge method into existing pre-trained models and continue to train the model on a multi-object dataset constructed from the open-sourced SA-1B dataset. To mitigate object confusion and reduce training costs, we propose an object quality score to estimate the image quality for the selection of high-quality training samples. Furthermore, our weighted-merge training framework can be employed on single-object generation when a single object has multiple reference images. The experiments verify that our method achieves superior performance to the state-of-the-arts on the Concept101 dataset and DreamBooth dataset of multi-object personalized image generation, and remarkably improves the performance on single-object personalized image generation. Our code is available at <a class="link-external link-https" href="https://github.com/hqhQAQ/MIP-Adapter" rel="external noopener nofollow">this https URL</a>.

MISC: Multi-Condition Injection and Spatially-Adaptive Compositing for Conditional Person Image Synthesis.

PersonaCraft: Personalized Full-Body Image Synthesis for Multiple Identities from Single References Using 3D-Model-Conditioned Diffusion

Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models

Person Image Synthesis in Arbitrary 3D Poses Based on Part Affinity Fields.

Multi3D: 3D-Aware Multimodal Image Synthesis

Conditional Image Repainting.

Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation

Where and Who? Automatic Semantic-Aware Person Composition

MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration

Utilizing Greedy Nature for Multimodal Conditional Image Synthesis in Transformers

Statistics Enhancement Generative Adversarial Networks for Diverse Conditional Image Synthesis

Scene Aware Person Image Generation through Global Contextual Conditioning

MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis

Appearance and Shape Based Image Synthesis by Conditional Variational Generative Adversarial Network

Composer: Creative and Controllable Image Synthesis with Composable Conditions

Conditional Image Synthesis with Diffusion Models: A Survey

Cocktail: Mixing Multi-Modality Control for Text-Conditional Image Generation

Unpaired semantic neural person image synthesis

On the Diversity of Conditional Image Synthesis with Semantic Layouts.

Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation

MCM: Multi-condition Motion Synthesis Framework