Abstract:Diffusion models have shown great promise in text-guided image style transfer, but there is a trade-off between style transformation and content preservation due to their stochastic nature. Existing methods require computationally expensive fine-tuning of diffusion models or additional neural network. To address this, here we propose a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks. By leveraging patch-wise contrastive loss between generated samples and original image embeddings in the pre-trained diffusion model, our method can generate images with the same semantic content as the source image in a zero-shot manner. Our approach outperforms existing methods while preserving content and requiring no additional training, not only for image style transfer but also for image-to-image translation and manipulation. Our experimental results validate the effectiveness of our proposed method.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: in text - guided image style transfer based on diffusion models, how to achieve efficient style transfer without losing content. Specifically, existing methods usually require additional fine - tuning or auxiliary networks to balance the trade - off between style transfer and content preservation, which leads to problems such as high computational costs and large training data requirements.
### Core problems of the paper:
1. **Trade - off between style transfer and content preservation**: Due to its randomness, the diffusion model is prone to losing the content information of the source image during style transfer.
2. **High computational cost**: Existing methods usually require additional fine - tuning of the diffusion model or the introduction of additional neural networks, which increases the computational complexity and training time.
3. **Lack of zero - shot ability**: Most existing methods rely on paired datasets for training and cannot be directly applied in unseen domains.
### Proposed solutions:
To address the above challenges, the authors propose a new method - **Zero - Shot Contrastive (ZeCon) loss** for image style transfer in diffusion models. The main features of this method are as follows:
- **No additional training required**: By leveraging the spatial information in the pre - trained diffusion model, style transfer can be achieved without additional training.
- **Zero - shot transfer**: It can be directly applied in unseen domains without the need for additional paired datasets.
- **Effective contrastive loss**: By introducing patch - wise contrastive loss between the generated samples and the original image embeddings, it is ensured that the generated image is semantically consistent with the source image.
### Formula representation:
1. **Contrastive loss formula**:
\[
\ell_{\text{ZeCon}} (\hat{x}_{0,t}, x_0) = \mathbb{E}_{x_0} \left[ \sum_l \sum_s \ell(\hat{z}_s^l, z_s^l, z_{S \setminus s}^l) \right]
\]
where $\hat{z}_l$ and $z_l$ represent the $l$-th layer feature maps from $\hat{x}_{0,t}$ and $x_0$ respectively, $s$ represents the spatial position in the feature map, and $\ell(ยท)$ is the cross - entropy loss function.
2. **Total content loss formula**:
\[
L_{\text{content}} = \ell_{\text{ZeCon}} (\hat{x}_{0,t}, x_0) + \ell_{\text{VGG}} (\hat{x}_{0,t}, x_0) + \ell_{\text{MSE}} (\hat{x}_{0,t}, x_0)
\]
3. **CLIP style loss formula**:
\[
\ell_{\text{CLIP}} = \ell_{\text{global}} (\hat{x}_{0,t}, p_{\text{target}}) + \ell_{\text{dir}} (\hat{x}_{0,t}, x_0, p_{\text{target}}, p_{\text{source}})
\]
where
\[
\ell_{\text{global}} (\hat{x}_{0,t}, p_{\text{target}}) = D_{\text{CLIP}} (\hat{x}_{0,t}, p_{\text{target}})
\]
\[
\ell_{\text{dir}} (\hat{x}_{0,t}, x_0, p_{\text{target}}, p_{\text{source}}) = 1 - \frac{\Delta I \cdot \Delta T}{\|\Delta I\| \|\Delta T\|}
\]