Abstract:Leveraging the visual priors of pre-trained text-to-image diffusion models offers a promising solution to enhance zero-shot generalization in dense prediction tasks. However, existing methods often uncritically use the original diffusion formulation, which may not be optimal due to the fundamental differences between dense prediction and image generation. In this paper, we provide a systemic analysis of the diffusion formulation for the dense prediction, focusing on both quality and efficiency. And we find that the original parameterization type for image generation, which learns to predict noise, is harmful for dense prediction; the multi-step noising/denoising diffusion process is also unnecessary and challenging to optimize. Based on these insights, we introduce Lotus, a diffusion-based visual foundation model with a simple yet effective adaptation protocol for dense prediction. Specifically, Lotus is trained to directly predict annotations instead of noise, thereby avoiding harmful variance. We also reformulate the diffusion process into a single-step procedure, simplifying optimization and significantly boosting inference speed. Additionally, we introduce a novel tuning strategy called detail preserver, which achieves more accurate and fine-grained predictions. Without scaling up the training data or model capacity, Lotus achieves SoTA performance in zero-shot depth and normal estimation across various datasets. It also enhances efficiency, being significantly faster than most existing diffusion-based methods. Lotus' superior quality and efficiency also enable a wide range of practical applications, such as joint estimation, single/multi-view 3D reconstruction, etc. Project page: <a class="link-external link-https" href="https://lotus3d.github.io/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the **zero - sample generalization problem in dense prediction tasks**, especially in two key geometry - aware tasks: depth estimation and normal estimation. Specifically, the paper points out: 1. **Limitations of existing methods**: - Existing diffusion - model - based methods usually directly inherit pre - trained text - to - image generation models (such as Stable Diffusion), but these models were originally designed for image generation and are not suitable for dense prediction tasks. - The standard multi - step diffusion process has problems such as optimization difficulties, high computational complexity, and easy propagation of prediction errors in dense prediction tasks. 2. **Research motivation**: - Dense prediction tasks require estimating pixel - level geometric properties from a single image, which requires a comprehensive understanding of the scene. - Although deep learning has made progress in dense prediction, its performance is limited by the quality, diversity, and scale of training data, resulting in poor zero - sample generalization ability. 3. **Proposed new method**: - The paper proposes a diffusion - based visual foundation model named **Lotus**, which is specifically optimized for dense prediction tasks. - Lotus solves the above problems through the following innovations: - **Directly predict annotations instead of noise**: Avoids the harmful variance brought by standard noise prediction. - **Single - step diffusion formula**: Simplifies the optimization process and significantly improves the inference speed. - **Detail Preserver**: By introducing a task - switching mechanism, it preserves the fine - grained details of the input image while generating dense annotations. 4. **Experimental verification**: - Experiments on multiple datasets show that Lotus achieves state - of - the - art performance in zero - sample depth and normal estimation tasks and is also more efficient than existing diffusion - model - based methods. In summary, this paper attempts to improve the accuracy and efficiency of dense prediction tasks by redesigning the parameterization type and optimization process of the diffusion model, so as to achieve better zero - sample generalization ability.

Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

DDP: Diffusion Model for Dense Visual Prediction

Exploiting Diffusion Prior for Generalizable Dense Prediction

PriorDiffusion: Leverage Language Prior in Diffusion Models for Monocular Depth Estimation

Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis

Successfully Applying Lottery Ticket Hypothesis to Diffusion Model

Learning on Less: Constraining Pre-trained Model Learning for Generalizable Diffusion-Generated Image Detection

What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?

Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models

FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes

Diffusion Models in Low-Level Vision: A Survey

Layered Rendering Diffusion Model for Zero-Shot Guided Image Synthesis

Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC

Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion

Diffusion Probabilistic Priors for Zero-Shot Low-Dose CT Image Denoising

Diffusion Models Need Visual Priors for Image Generation

Training-free Prior Guided Diffusion Model for Zero-Reference Low-Light Image Enhancement

Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models

Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation