Abstract:Diffusion-based generative models have achieved remarkable success in image generation. Their guidance formulation allows an external model to plug-and-play control the generation process for various tasks without finetuning the diffusion model. However, the direct use of publicly available off-the-shelf models for guidance fails due to their poor performance on noisy inputs. For that, the existing practice is to fine-tune the guidance models with labeled data corrupted with noises. In this paper, we argue that this practice has limitations in two aspects: (1) performing on inputs with extremely various noises is too hard for a single guidance model; (2) collecting labeled datasets hinders scaling up for various tasks. To tackle the limitations, we propose a novel strategy that leverages multiple experts where each expert is specialized in a particular noise range and guides the reverse process of the diffusion at its corresponding timesteps. However, as it is infeasible to manage multiple networks and utilize labeled data, we present a practical guidance framework termed Practical Plug-And-Play (PPAP), which leverages parameter-efficient fine-tuning and data-free knowledge transfer. We exhaustively conduct ImageNet class conditional generation experiments to show that our method can successfully guide diffusion with small trainable parameters and no labeled data. Finally, we show that image classifiers, depth estimators, and semantic segmentation models can guide publicly available GLIDE through our framework in a plug-and-play manner. Our code is available at <a class="link-external link-https" href="https://github.com/riiid/PPAP" rel="external noopener nofollow">this https URL</a>.

Drag-guided diffusion models for vehicle image generation

Data-Driven Car Drag Prediction With Depth and Normal Renderings

Surrogate Modeling of Car Drag Coefficient with Depth and Normal Renderings

Towards Practical Plug-and-Play Diffusion Models

DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models

Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation

Boosting Visual Fidelity in Driving Simulations through Diffusion Models

DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models

DragTraffic: Interactive and Controllable Traffic Scene Generation for Autonomous Driving

DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving

Interpretable ODE-style Generative Diffusion Model via Force Field Construction

Dynamic Negative Guidance of Diffusion Models

3DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models

PhysDiff: Physics-Guided Human Motion Diffusion Model

Guided Diffusion from Self-Supervised Diffusion Features

Physics-Informed Diffusion Models

AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing

Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

Accelerated Image-Aware Generative Diffusion Modeling

VehicleSDF: A 3D generative model for constrained engineering design via surrogate modeling

GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models