StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On

Jeongho Kim,Gyojung Gu,Minho Park,Sunghyun Park,Jaegul Choo

2023-12-04

Abstract:Given a clothing image and a person image, an image-based virtual try-on aims to generate a customized image that appears natural and accurately reflects the characteristics of the clothing image. In this work, we aim to expand the applicability of the pre-trained diffusion model so that it can be utilized independently for the virtual try-on task.The main challenge is to preserve the clothing details while effectively utilizing the robust generative capability of the pre-trained model. In order to tackle these issues, we propose StableVITON, learning the semantic correspondence between the clothing and the human body within the latent space of the pre-trained diffusion model in an end-to-end manner. Our proposed zero cross-attention blocks not only preserve the clothing details by learning the semantic correspondence but also generate high-fidelity images by utilizing the inherent knowledge of the pre-trained model in the warping process. Through our proposed novel attention total variation loss and applying augmentation, we achieve the sharp attention map, resulting in a more precise representation of clothing details. StableVITON outperforms the baselines in qualitative and quantitative evaluation, showing promising quality in arbitrary person images. Our code is available at <a class="link-external link-https" href="https://github.com/rlawjdghek/StableVITON" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve several key problems in the Virtual Try - On (VTO) task:

1. **Preserving clothing details**: In virtual try - on, how to accurately reflect the details of clothing in the generated image is a challenge. Existing methods often have difficulty in maintaining clothing details when dealing with arbitrary person images with complex backgrounds.

2. **Fully utilizing the capabilities of pre - trained diffusion models**: Pre - trained diffusion models have powerful generation capabilities, but when applying them to the virtual try - on task, how to effectively utilize the knowledge of these models is a difficult problem. Existing methods usually rely on external alignment networks to handle the semantic correspondence between clothing and the human body, which limits the performance of the model.

3. **Improving the generalization ability of the model**: The performance of existing virtual try - on methods will significantly decline when dealing with person images from different datasets or with complex backgrounds. Therefore, how to improve the generalization ability of the model in different environments is also an important issue.

To solve these problems, the authors propose the **StableVITON** model, whose main innovations include:

- **End - to - end learning of semantic correspondence**: By learning the semantic correspondence between clothing and the human body in the latent space of the pre - trained diffusion model, the dependence on external alignment networks is avoided.

- **Zero Cross - Attention Block**: A new mechanism is introduced to perform fine - grained feature alignment in the latent space, thereby better preserving clothing details and generating high - quality images.

- **Attention Total Variation Loss**: A new loss function is proposed to sharpen the attention map and ensure the accuracy of clothing details in the generated image.

Through these improvements, StableVITON outperforms existing virtual try - on methods in both qualitative and quantitative evaluations and shows better generalization ability when dealing with different datasets.

### Formula Summary

1. **Forward diffusion process**:
\[
q(z_t|z_0)=\mathcal{N}(z_t; \sqrt{\bar{\alpha}_t}z_0,(1 - \bar{\alpha}_t)I)
\]
where \(t\in\{1,\ldots,T\}\), \(\alpha_t := 1-\beta_t\), \(\bar{\alpha}_t:=\prod_{s = 1}^t\alpha_s\).

2. **Simplified objective function**:
\[
L_{LDM}=\mathbb{E}_{x,y,\epsilon\sim\mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_\theta(z_t,t,\tau_\theta(y))\|^2_2\right]
\]

3. **Attention center coordinate map**:
\[
F_{ij}^n=\frac{1}{h_k w_k}\sum_{k = 1}^{h_k}\sum_{l = 1}^{w_k}(A_{ijkl}\odot G_{kln})
\]
where \(A\in\mathbb{R}^{H_q\times W_q\times h_k\times w_k}\), \(G\in[- 1,1]^{h_k\times w_k\times2}\) is the normalized two - dimensional coordinate, and \(\odot\) represents element - wise multiplication.

4. **Attention Total Variation Loss**:
\[
L_{ATV}=\|\nabla(F\odot M)\|_1
\]
where \(M\in\{0,1\}^{H_q\times W_q}\) is the ground - truth clothing mask.

5. **Fine - tuning loss function**: