Abstract:Due to their text-to-image synthesis feature, diffusion models have recently seen a rise in visual perception tasks, such as depth estimation. The lack of good-quality datasets makes the extraction of a fine-grain semantic context challenging for the diffusion models. The semantic context with fewer details further worsens the process of creating effective text embeddings that will be used as input for diffusion models. In this paper, we propose a novel EDADepth, an enhanced data augmentation method to estimate monocular depth without using additional training data. We use Swin2SR, a super-resolution model, to enhance the quality of input images. We employ the BEiT pre-trained semantic segmentation model for better extraction of text embeddings. We use BLIP-2 tokenizer to generate tokens from these text embeddings. The novelty of our approach is the introduction of Swin2SR, the BEiT model, and the BLIP-2 tokenizer in the diffusion-based pipeline for the monocular depth estimation. Our model achieves state-of-the-art results (SOTA) on the delta3 metric on NYUv2 and KITTI datasets. It also achieves results comparable to those of the SOTA models in the RMSE and REL metrics. Finally, we also show improvements in the visualization of the estimated depth compared to the SOTA diffusion-based monocular depth estimation models. Code: <a class="link-external link-https" href="https://github.com/edadepthmde/EDADepth_ICMLA" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are several key challenges in monocular depth estimation (MDE): 1. **Lack of high - quality datasets**: There are few existing high - quality datasets, making it difficult to extract fine - grained semantic context. This further deteriorates the process of creating effective text embeddings for diffusion models. 2. **Impact of low - quality input images**: Low - quality input images may lead to incomplete or incorrect semantic information, thus affecting the accuracy of depth estimation. 3. **Limitations of existing methods**: Traditional methods, such as those based on traditional cues and shadows in single - view images, perform poorly in the face of changing lighting conditions and when precise camera calibration is required. To this end, the paper proposes a new enhanced data augmentation method - EDADepth, which aims to improve monocular depth estimation in the following ways: - **Use the pre - trained Swin2SR model** to enhance the quality of input images, thereby improving the estimation of depth maps. - **Introduce the BEiT semantic segmentation model** to extract more detailed text embeddings in order to capture the semantic context of input images. - **Adopt the BLIP - 2 tokenizer** to generate text embedding tokens, improving the quality and accuracy of text embeddings. These improvements enable EDADepth to achieve state - of - the - art (SOTA) results on the δ3 metric in the NYUv2 and KITTI datasets, and also perform well on other metrics such as RMSE and REL. In addition, EDADepth also outperforms existing diffusion - model - based monocular depth estimation methods in terms of visualization effects. ### Formula Representation The formulas involved in the paper include: - Forward process of the diffusion model: \[ z_t\sim\mathcal{N}(\sqrt{\alpha_t}z_{t - 1},(1-\alpha_t)I) \] where \(z_t\) is a random variable at time \(t\), \(\alpha_t\) is a fixed noise scheduling coefficient, and \(\mathcal{N}(z,\mu,\sigma)\) represents the normal distribution. - Loss function of the diffusion model: \[ L_{LDM}:=\mathbb{E}_{E(x),y,\epsilon\sim\mathcal{N}(0,1),t}\left\|\epsilon-\epsilon_\theta(z_t,t,\tau_\theta(y))\right\|_2^2 \] where \(z_t\) is calculated by formula (1). - Probability distribution modeling: \[ p_\lambda(y|x,T)=p_\lambda4(y|z_0)p_\lambda3(z_0|z_t,C)p_\lambda2(z_t|x)p_\lambda1(C,x) \] where \(T\) represents the text embedding obtained from the BEiT model, and \(C\) represents the context information. Through these improvements, EDADepth significantly improves the accuracy and robustness of monocular depth estimation.

EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Semantic-guided Depth Completion from Monocular Images and 4D Radar Data

Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

Monocular Depth Estimation using Diffusion Models

Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation

ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation

PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimage

PriorDiffusion: Leverage Language Prior in Diffusion Models for Monocular Depth Estimation

Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions

DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimation

Towards Better Data Exploitation in Self-Supervised Monocular Depth Estimation

BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation

Edge-Enhanced Dual-Stream Perception Network for Monocular Depth Estimation

Unsupervised Monocular Depth Estimation Based on Hierarchical Feature-Guided Diffusion

Boosting Monocular Depth Estimation with Sparse Guided Points

SwinDepth: Unsupervised Depth Estimation using Monocular Sequences via Swin Transformer and Densely Cascaded Network

Enhanced Monocular Depth Estimation: A CNN Integrating Semantic Segmentation Embedding And Vanishing Point Detection

Depth-guided Texture Diffusion for Image Semantic Segmentation

Monocular Depth Estimation Using Cues Inspired by Biological Vision Systems

FA-Depth: Toward Fast and Accurate Self-supervised Monocular Depth Estimation