Abstract:We propose a diffusion model-based approach, FloAtControlNet to generate cinemagraphs composed of animations of human clothing. We focus on human clothing like dresses, skirts and pants. The input to our model is a text prompt depicting the type of clothing and the texture of clothing like leopard, striped, or plain, and a sequence of normal maps that capture the underlying animation that we desire in the output. The backbone of our method is a normal-map conditioned ControlNet which is operated in a training-free regime. The key observation is that the underlying animation is embedded in the flow of the normal maps. We utilize the flow thus obtained to manipulate the self-attention maps of appropriate layers. Specifically, the self-attention maps of a particular layer and frame are recomputed as a linear combination of itself and the self-attention maps of the same layer and the previous frame, warped by the flow on the normal maps of the two frames. We show that manipulating the self-attention maps greatly enhances the quality of the clothing animation, making it look more natural as well as suppressing the background artifacts. Through extensive experiments, we show that the method proposed beats all baselines both qualitatively in terms of visual results and user study. Specifically, our method is able to alleviate the background flickering that exists in other diffusion model-based baselines that we consider. In addition, we show that our method beats all baselines in terms of RMSE and PSNR computed using the input normal map sequences and the normal map sequences obtained from the output RGB frames. Further, we show that well-established evaluation metrics like LPIPS, SSIM, and CLIP scores that are generally for visual quality are not necessarily suitable for capturing the subtle motions in human clothing animations.

Appearance flow estimation for online virtual clothing warping via optimal feature linear assignment

Style-Based Global Appearance Flow for Virtual Try-On

Continuous Perception Garment Classification Based on Optical Flow Variation.

ClothFormer:Taming Video Virtual Try-on in All Module

C2F-CFN: Coarse-to-Fine ClothFlow Network for High-Fidelity Virtual Try-On

Single Stage Warped Cloth Learning and Semantic-Contextual Attention Feature Fusion for Virtual TryOn

VTON-SCFA: A Virtual Try-On Network Based on the Semantic Constraints and Flow Alignment

ZFlow: Gated Appearance Flow-based Virtual Try-on with 3D Priors

Limb-Aware Virtual Try-On Network with Progressive Clothing Warping

Toward Realistic Virtual Try-on Through Landmark Guided Shape Matching

GP-VTON: Towards General Purpose Virtual Try-on via Collaborative Local-Flow Global-Parsing Learning

Dressing in the Wild by Watching Dance Videos

KF-VTON: Keypoints-Driven Flow Based Virtual Try-On Network

Three stages of 3D virtual try-on network with appearance flow and shape field

Example-based Real-time Clothing Synthesis for Virtual Agents

GraVITON: Graph based garment warping with attention guided inversion for Virtual-tryon

Toward Accurate and Realistic Virtual Try-on Through Shape Matching and Multiple Warps

AgentDress: Realtime Clothing Synthesis for Virtual Agents using Plausible Deformations

Recurrent Appearance Flow for Occlusion-Free Virtual Try-On

WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on

FloAt: Flow Warping of Self-Attention for Clothing Animation Generation