Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model

Han Lin,Jaemin Cho,Abhay Zala,Mohit Bansal

2024-05-25

Abstract:ControlNets are widely used for adding spatial control to text-to-image diffusion models with different conditions, such as depth maps, scribbles/sketches, and human poses. However, when it comes to controllable video generation, ControlNets cannot be directly integrated into new backbones due to feature space mismatches, and training ControlNets for new backbones can be a significant burden for many users. Furthermore, applying ControlNets independently to different frames cannot effectively maintain object temporal consistency. To address these challenges, we introduce Ctrl-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffusion model through the adaptation of pretrained ControlNets. Ctrl-Adapter offers strong and diverse capabilities, including image and video control, sparse-frame video control, fine-grained patch-level multi-condition control (via an MoE router), zero-shot adaptation to unseen conditions, and supports a variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided motion control. With six diverse U-Net/DiT-based image/video diffusion models (SDXL, PixArt-$\alpha$, I2VGen-XL, SVD, Latte, Hotshot-XL), Ctrl-Adapter matches the performance of pretrained ControlNets on COCO and achieves the state-of-the-art on DAVIS 2017 with significantly lower computation (< 10 GPU hours).

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

This paper presents a solution to the problems encountered by existing ControlNets when adapting to new image/video diffusion models. Existing ControlNets are popular for text-to-image generation, but they cannot be directly applied to new video generation models due to feature space mismatch, and training a new ControlNet incurs high computational costs. Moreover, applying ControlNet separately to different video frames fails to effectively maintain temporal consistency of objects.

To address these issues, the paper proposes an efficient and general framework called CTRL-Adapter. CTRL-Adapter adapts pre-trained ControlNets, enabling any image/video diffusion model to incorporate multiple controls for better temporal consistency. It includes spatial convolution, temporal convolution, spatial attention, and temporal attention modules to integrate ControlNet features. Additionally, it supports zero-shot adaptation to unseen conditions as well as multi-condition control.

The paper demonstrates the application of CTRL-Adapter on various diffusion models such as SDXL, PixArt-α, I2VGen-XL, SVD, Latte, and Hotshot-XL. It performs comparably to pre-trained ControlNet on the COCO dataset and achieves state-of-the-art results on the DAVIS 2017 dataset while significantly reducing computational costs (less than 10 GPU hours). It can also be applied to downstream tasks such as video editing, video style transfer, and text-guided object motion control.

Through extensive experiments and analysis, the paper proves the effectiveness of CTRL-Adapter, particularly in handling multi-condition control and sparse-frame video control, outperforming other methods.