Abstract:As Classifier-Free Guidance (CFG) has proven effective in conditional diffusion model sampling for improved condition alignment, many applications use a negated CFG term to filter out unwanted features from samples. However, simply negating CFG guidance creates an inverted probability distribution, often distorting samples away from the marginal distribution. Inspired by recent advances in conditional diffusion models for inverse problems, here we present a novel method to enhance negative CFG guidance using contrastive loss. Specifically, our guidance term aligns or repels the denoising direction based on the given condition through contrastive loss, achieving a nearly identical guiding direction to traditional CFG for positive guidance while overcoming the limitations of existing negative guidance methods. Experimental results demonstrate that our approach effectively removes undesirable concepts while maintaining sample quality across diverse scenarios, from simple class conditions to complex and overlapping text prompts.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the quality degradation and sample deviation issues encountered when using negative guidance in diffusion models. Specifically, traditional negative guidance methods such as simple negated Classifier - Free Guidance (nCFG) and Dynamic Negative Guidance (DNG), when avoiding certain unwanted concepts, often reduce sample quality or cause the sample distribution to deviate from the support region of the original data distribution. These problems are particularly prominent when dealing with complex conditions (such as overlapping conditions in text - to - image generation tasks). To overcome these limitations, the authors propose Contrastive Classifier - Free Guidance (CCFG), which optimizes the denoising direction by introducing contrastive loss, thereby maintaining sample quality while avoiding unwanted concepts. CCFG ensures a more precise and stable response to given conditions during the sampling process by automatically adjusting the attractive and repulsive forces. ### Main contributions of the paper 1. **Proposing the CCFG method**: By optimizing the denoising direction with contrastive loss, the model can maintain the quality of samples while avoiding unwanted concepts. 2. **Theoretical analysis**: It analyzes in detail the limitations of traditional negative guidance methods (such as nCFG and DNG), and shows through theoretical derivation how CCFG overcomes these limitations. 3. **Experimental verification**: Experiments were carried out on multiple datasets, including MNIST, CIFAR10, and StableDiffusion 1.5, verifying the effectiveness and superiority of CCFG. ### Experimental results - **Performance on class - conditional models**: On the MNIST and CIFAR10 datasets, CCFG maintains a high sample quality (FID score) while reducing the error rate. - **Performance on text - to - image models**: On StableDiffusion 1.5, CCFG can more effectively remove unwanted concepts, while maintaining a high degree of consistency with positive prompts, and the image quality is better than other methods. ### Discussion - **Analysis of guidance scales**: Through contrastive loss, CCFG exhibits stable linear behavior during positive guidance and can gradually weaken the guidance strength during negative guidance, avoiding the instability and excessive repulsion problems in traditional methods. - **Performance in different scenarios**: Although nCFG may perform reasonably well in simple tasks, its performance in complex conditions is poor. CCFG can maintain stable and high - quality sample generation in various scenarios by dynamically adjusting the guidance strength. In conclusion, this paper effectively solves the problems of negative guidance in diffusion models by proposing the CCFG method, providing a new solution for the safety and controllability of generative models.

Contrastive CFG: Improving CFG in Diffusion Models by Contrasting Positive and Negative Concepts

CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models

Rectified Diffusion Guidance for Conditional Generation

Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models

Classifier-Free Guidance is a Predictor-Corrector

Debiased Graph Contrastive Learning.

No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models

Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance

Inner Classifier-Free Guidance and Its Taylor Expansion for Diffusion Models

EP-CFG: Energy-Preserving Classifier-Free Guidance

Dynamic Negative Guidance of Diffusion Models

Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models

Negative as Positive: Enhancing Out-of-distribution Generalization for Graph Contrastive Learning

Diffusion Model Conditioning on Gaussian Mixture Model and Negative Gaussian Mixture Gradient

Null-text Guidance in Diffusion Models is Secretly a Cartoon-style Creator

SafeCFG: Redirecting Harmful Classifier-Free Guidance for Safe Generation

Guidance with Spherical Gaussian Constraint for Conditional Diffusion

ReGCL: Rethinking Message Passing in Graph Contrastive Learning

Adaptive Negative Representations for Graph Contrastive Learning

Classifier-Free Diffusion Guidance