Abstract:Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. Nevertheless, prevailing text-guided visual diffusion models primarily focus on incorporating text-visual relationships exclusively into the reverse process, often disregarding their relevance in the forward process. This inconsistency between forward and reverse processes may limit the precise conveyance of textual semantics in visual synthesis results. To address this issue, we propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample into forward and reverse processes. We propagate this context to all timesteps in the two processes to adapt their trajectories, thereby facilitating cross-modal conditional modeling. We generalize our contextualized diffusion to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing. In each task, our ContextDiff achieves new state-of-the-art performance, significantly enhancing the semantic alignment between text condition and generated samples, as evidenced by quantitative and qualitative evaluations. Our code is available at <a class="link-external link-https" href="https://github.com/YangLing0818/ContextDiff" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the inconsistency in the utilization of text conditions in the forward and reverse processes of existing text - guided visual diffusion models, as well as the fact that these models usually overlook the cross - modal context between text and visual modalities. This inconsistency and oversight limit the precise expression of text semantics in visual synthesis results. To address these issues, the paper proposes a new cross - modal context diffusion model (CONTEXT DIFF), which optimizes the diffusion trajectory by incorporating cross - modal interactions and alignments between text conditions and visual samples into the forward and reverse processes, thereby improving the semantic alignment in text - guided image and video generation tasks. Specifically, the contributions of the paper include: 1. **First Proposal**: Propose CONTEXT DIFF for the first time, considering cross - modal interaction as a context - aware trajectory adapter to contextualize the forward and sampling processes in text - guided visual diffusion models. 2. **Theoretical Generalization**: Generalize the contextualized diffusion model to DDPMs and DDIMs, and provide detailed theoretical derivations to support cross - modal visual generation and editing tasks. 3. **Performance Improvement**: Achieve new state - of - the - art performance in text - to - image generation and text - to - video editing tasks, and consistently demonstrate the superiority of CONTEXT DIFF over existing diffusion models through quantitative and qualitative comparisons. The paper verifies the effectiveness of its method through experiments. In particular, in text - to - image generation and text - to - video editing tasks, CONTEXT DIFF not only improves semantic alignment but also maintains the structural information of the source video, demonstrating its significant advantage in cross - modal understanding.

Contextualized Diffusion Models for Text-Guided Image and Video Generation

Cross-Modal Contextualized Diffusion Models for Text-Guided Visual Generation and Editing

In-Context Learning Unlocked for Diffusion Models

Context Diffusion: In-Context Aware Image Generation

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

GlyphDiffusion: Text Generation as Image Generation

Improving Diffusion-Based Image Synthesis with Context Prediction

Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

Conditional Text Image Generation with Diffusion Models

DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models

Versatile Diffusion: Text, Images and Variations All in One Diffusion Model

Improving In-Context Learning in Diffusion Models with Visual Context-Modulated Prompts

Unleashing Text-to-Image Diffusion Models for Visual Perception

DiffusionRet: Generative Text-Video Retrieval with Diffusion Model

DiffUTE: Universal Text Editing Diffusion Model

Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation

DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

Unified Discrete Diffusion for Simultaneous Vision-Language Generation

UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models