Unsupervised Learning Based Focal Stack Camera Depth Estimation

Zhengyu Huang,Weizhi Du,Theodore B. Norris

DOI: https://doi.org/10.48550/arXiv.2203.07904

2022-03-14

Image and Video Processing

Abstract:We propose an unsupervised deep learning based method to estimate depth from focal stack camera images. On the NYU-v2 dataset, our method achieves much better depth estimation accuracy compared to single-image based methods.

What problem does this paper attempt to address?

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Jiafeng Shen,Kaiwei Wang,Kailun Yang,Kaite Xiang,Lei Fei,Xinxin Hu,Huabing Li,Hao Chen

DOI: https://doi.org/10.1117/12.2532666

2019-01-01

Abstract:In recent years, with the vigorous development of artificial intelligence and autonomous driving technology, the importance of scene perception technology is increasing. Unsupervised deep learning based methods have demonstrated a certain level of robustness and accuracy in some challenging scenes. By inferring depth from a single input image without any ground truth label, a lot of time and resources can be saved. However, unsupervised depth estimation has defects in robustness and accuracy under complex environment which could be improved by modifying network structure and incorporating other modal information. In this paper, we propose an unsupervised, monocular depth estimation network achieving high speed and accuracy, and a learning framework with our depth estimation network to improve depth performance by incorporating transformed images across different modalities. The depth estimator is an encoder-decoder network to generate the multi-scale dense depth map. The sub-pixel convolutional layer is adopted to obtain depth super-resolution by replacing the up-sample branches. The cross-modal depth estimation using near-infrared image and RGB image enhances the performance of depth estimation than pure RGB image. The training mode is to transfer both images to the same modality and then carry out super-resolved depth estimation for each stereo camera pair. Compared with the initial results of depth estimation using only RGB images, the experiment verifies that our depth estimation network with the cross-modal fusion system designed in this paper achieves better performance on public datasets and a multi-modal dataset collected by our stereo vision sensor.
Monocular Depth Estimation Based on Unsupervised Learning

Wan Liu,Yan Sun,XuCheng Wang,Lin Yang,Zhenrong Zheng

DOI: https://doi.org/10.1117/12.2537957

2019-01-01

Abstract:Due to the low cost and easy deployment, the depth estimation of monocular cameras has always attracted attention of researchers. As good performance based on deep learning technology in depth estimation, more and more training models has emerged for depth estimation. Most existing works have required very promising results that belongs to supervised learning methods, but corresponding ground truth depth data for training is inevitable that makes training complicated. To overcome this limitation, an unsupervised learning framework is used for monocular depth estimation from videos, which contains depth map and pose network. In this paper, better results can be achieved by optimizing training models and improving training loss. Besides, training and evaluation data is based on standard dataset KITTI(Karlsruhe Institute of Technology and Toyota Institute of Technology). In the end, the results are shown through comparing with different training models used in this paper.
Least Square Estimation Network for Depth Completion

Xianze Fang,Zexi Chen,Yunkai Wang,Yue Wang,Rong Xiong

DOI: https://doi.org/10.48550/arXiv.2203.03317

2022-01-01

Abstract: Depth completion is a fundamental task in computer vision and robotics research. Many previous works complete the dense depth map with neural networks directly but most of them are non-interpretable and can not generalize to different situations well. In this paper, we propose an effective image representation method for depth completion tasks. The input of our system is a monocular camera frame and the synchronous sparse depth map. The output of our system is a dense per-pixel depth map of the frame. First we use a neural network to transform each pixel into a feature vector, which we call base functions. Then we pick out the known pixels' base functions and their depth values. We use a linear least square algorithm to fit the base functions and the depth values. Then we get the weights estimated from the least square algorithm. Finally, we apply the weights to the whole image and predict the final depth map. Our method is interpretable so it can generalize well. Experiments show that our results beat the state-of-the-art on NYU-Depth-V2 dataset both in accuracy and runtime. Moreover, experiments show that our method can generalize well on different numbers of sparse points and different datasets.
Monocular Depth Estimation Based on Multi-Scale Graph Convolution Networks

Junwei Fu,Jun Liang,Ziyang Wang

DOI: https://doi.org/10.1109/access.2019.2961606

IF: 3.9

2019-01-01

IEEE Access

Abstract:Monocular depth estimation is a foundation task of three-dimensional (3D) reconstruction which is used to improve the accuracy of environment perception. Because of the simpler hardware requirement, it is more suitable than other multi-view methods. In this study, a new monocular depth estimation algorithm based on graph convolution network (GCN) is proposed. The pixel-wise depth relationship is introduced into conventional convolution neural network (CNN) to make up the disadvantage of processing non-Euclidian data. And the remaining depth topological graph information on the spatial latent variables are extracted based on a multi-scale reconstruction strategy. The final results on NYU-v2 depth dataset and KITTI depth dataset demonstrate that our algorithm improves the quality of monocular depth estimation, especially there are several little objects coexisting in the scenes.
FocDepthFormer: Transformer with latent LSTM for Depth Estimation from Focal Stack

Xueyang Kang,Fengze Han,Abdur R. Fayjie,Patrick Vandewalle,Kourosh Khoshelham,Dong Gong

2024-11-25

Abstract:Most existing methods for depth estimation from a focal stack of images employ convolutional neural networks (CNNs) using 2D or 3D convolutions over a fixed set of images. However, their effectiveness is constrained by the local properties of CNN kernels, which restricts them to process only focal stacks of fixed number of images during both training and inference. This limitation hampers their ability to generalize to stacks of arbitrary lengths. To overcome these limitations, we present a novel Transformer-based network, FocDepthFormer, which integrates a Transformer with an LSTM module and a CNN decoder. The Transformer's self-attention mechanism allows for the learning of more informative spatial features by implicitly performing non-local cross-referencing. The LSTM module is designed to integrate representations across image stacks of varying lengths. Additionally, we employ multi-scale convolutional kernels in an early-stage encoder to capture low-level features at different degrees of focus/defocus. By incorporating the LSTM, FocDepthFormer can be pre-trained on large-scale monocular RGB depth estimation datasets, improving visual pattern learning and reducing reliance on difficult-to-obtain focal stack data. Extensive experiments on diverse focal stack benchmark datasets demonstrate that our model outperforms state-of-the-art approaches across multiple evaluation metrics.

Computer Vision and Pattern Recognition,Artificial Intelligence,Image and Video Processing
FS-Depth: Focal-and-Scale Depth Estimation from a Single Image in Unseen Indoor Scene

Chengrui Wei,Meng Yang,Lei He,Nanning Zheng

2023-07-27

Abstract:It has long been an ill-posed problem to predict absolute depth maps from single images in real (unseen) indoor scenes. We observe that it is essentially due to not only the scale-ambiguous problem but also the focal-ambiguous problem that decreases the generalization ability of monocular depth estimation. That is, images may be captured by cameras of different focal lengths in scenes of different scales. In this paper, we develop a focal-and-scale depth estimation model to well learn absolute depth maps from single images in unseen indoor scenes. First, a relative depth estimation network is adopted to learn relative depths from single images with diverse scales/semantics. Second, multi-scale features are generated by mapping a single focal length value to focal length features and concatenating them with intermediate features of different scales in relative depth estimation. Finally, relative depths and multi-scale features are jointly fed into an absolute depth estimation network. In addition, a new pipeline is developed to augment the diversity of focal lengths of public datasets, which are often captured with cameras of the same or similar focal lengths. Our model is trained on augmented NYUDv2 and tested on three unseen datasets. Our model considerably improves the generalization ability of depth estimation by 41%/13% (RMSE) with/without data augmentation compared with five recent SOTAs and well alleviates the deformation problem in 3D reconstruction. Notably, our model well maintains the accuracy of depth estimation on original NYUDv2.

Computer Vision and Pattern Recognition
Depth Estimation from Monocular Images Using Dilated Convolution and Uncertainty Learning.

Haojie Ma,Yinzhang Ding,Lianghao Wang,Ming Zhang,Dongxiao Li

DOI: https://doi.org/10.1007/978-3-030-00767-6_2

2018-01-01

Abstract:Depth cues are vital in many challenging computer vision tasks. In this paper, we address the problem of dense depth prediction from a single RGB image. Compared with stereo depth estimation, sensing the depth of a scene from monocular images is much more difficult and ambiguous because the epipolar geometry constraints cannot be exploited. In addition, the value of the scale is often unknown in monocular depth prediction. To facilitate an accurate single-view depth prediction, we introduce dilated convolution to capture multi-scale contextual information and then present a deep convolutional neural network. To improve the robustness of the system, we estimate the uncertainty of noisy data by modelling such uncertainty in a new loss function. The experiment results show that the proposed approach outperforms the previous state-of-the-art methods in depth estimation tasks.
Unsupervised Scale-Consistent Depth Learning from Video

Jia-Wang Bian,Huangying Zhan,Naiyan Wang,Zhichao Li,Le Zhang,Chunhua Shen,Ming-Ming Cheng,Ian Reid

DOI: https://doi.org/10.1007/s11263-021-01484-6

IF: 13.369

2021-06-18

International Journal of Computer Vision

Abstract:We propose a monocular depth estimation method <i>SC-Depth</i>, which requires only unlabelled videos for training and enables the scale-consistent prediction at inference time. Our contributions include: (i) we propose a geometry consistency loss, which penalizes the inconsistency of predicted depths between adjacent views; (ii) we propose a self-discovered mask to automatically localize moving objects that violate the underlying static scene assumption and cause noisy signals during training; (iii) we demonstrate the efficacy of each component with a detailed ablation study and show high-quality depth estimation results in both KITTI and NYUv2 datasets. Moreover, thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into ORB-SLAM2 system for more robust and accurate tracking. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training. Finally, we provide several demos for qualitative evaluation. The source code is released on GitHub.

computer science, artificial intelligence
Dynamic Fusion Network For Light Field Depth Estimation

Yongri Piao,Yukun Zhang,Miao Zhang,Xinxin Ji

DOI: https://doi.org/10.48550/arXiv.2104.05969

2021-04-13

Computer Vision and Pattern Recognition

Abstract:Focus based methods have shown promising results for the task of depth estimation. However, most existing focus based depth estimation approaches depend on maximal sharpness of the focal stack. Out of focus information in the focal stack poses challenges for this task. In this paper, we propose a dynamically multi modal learning strategy which incorporates RGB data and the focal stack in our framework. Our goal is to deeply excavate the spatial correlation in the focal stack by designing the spatial correlation perception module and dynamically fuse multi modal information between RGB data and the focal stack in a adaptive way by designing the multi modal dynamic fusion module. The success of our method is demonstrated by achieving the state of the art performance on two datasets. Furthermore, we test our network on a set of different focused images generated by a smart phone camera to prove that the proposed method not only broke the limitation of only using light field data, but also open a path toward practical applications of depth estimation on common consumer level cameras data.
Fully Self-Supervised Depth Estimation from Defocus Clue

Haozhe Si,Bin Zhao,Dong Wang,Yunpeng Gao,Mulin Chen,Zhigang Wang,Xuelong Li

2023-03-27

Abstract:Depth-from-defocus (DFD), modeling the relationship between depth and defocus pattern in images, has demonstrated promising performance in depth estimation. Recently, several self-supervised works try to overcome the difficulties in acquiring accurate depth ground-truth. However, they depend on the all-in-focus (AIF) images, which cannot be captured in real-world scenarios. Such limitation discourages the applications of DFD methods. To tackle this issue, we propose a completely self-supervised framework that estimates depth purely from a sparse focal stack. We show that our framework circumvents the needs for the depth and AIF image ground-truth, and receives superior predictions, thus closing the gap between the theoretical success of DFD works and their applications in the real world. In particular, we propose (i) a more realistic setting for DFD tasks, where no depth or AIF image ground-truth is available; (ii) a novel self-supervision framework that provides reliable predictions of depth and AIF image under the challenging setting. The proposed framework uses a neural model to predict the depth and AIF image, and utilizes an optical model to validate and refine the prediction. We verify our framework on three benchmark datasets with rendered focal stacks and real focal stacks. Qualitative and quantitative evaluations show that our method provides a strong baseline for self-supervised DFD tasks.

Image and Video Processing,Computer Vision and Pattern Recognition
Monocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference

Bo Li,Yuchao Dai,Mingyi He

DOI: https://doi.org/10.1016/j.patcog.2018.05.029

IF: 8

2018-11-01

Pattern Recognition

Abstract:Monocular depth estimation is very challenging in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks, the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contrast to the regression-based formulation. In this way, we could build upon the recent progress in dense labeling such as semantic segmentation. Second, we fuse different side-outputs from our front-end dilated convolutional neural network in a hierarchical way to exploit the multi-scale depth cues for monocular depth estimation, which is critical in achieving scale-aware depth estimation. Third, we propose to utilize soft-weighted-sum inference instead of the hard-max inference, transforming the discretized depth scores to continuous depth values. Thus, we reduce the influence of quantization error and improve the robustness of our method. Extensive experiments have been conducted on the Make3D, NYU v2, and KITTI datasets and superior performance have been achieved on NYU v2 and KITTI datasets compared with current state-of-the-art methods, which shows the superiority of our method. Furthermore, experiments on the NYU v2 dataset reveal that our classification based model is able to learn the probability distribution of depth.

computer science, artificial intelligence,engineering, electrical & electronic
Self-supervised multi-frame depth estimation with visual-inertial pose transformer and monocular guidance

Xiang Wang,Haonan Luo,Zihang Wang,Jin Zheng,Xiao Bai

DOI: https://doi.org/10.1016/j.inffus.2024.102363

IF: 18.6

2024-03-18

Information Fusion

Abstract:Self-supervised monocular depth estimation has been a popular topic since it does not need labor-intensive depth ground truth collection. However, the accuracy of monocular network is limited as it can only utilize context provided in the single image, ignoring the geometric clues resided in videos. Most recently, multi-frame depth networks are introduced to the self-supervised depth learning framework to ameliorate monocular depth, which explicitly encode the geometric information via pairwise cost volume construction. In this paper, we address two main issues that affect the cost volume construction and thus the multi-frame depth estimation. First, camera pose estimation, which determines the epipolar geometry in cost volume construction but has rarely been addressed, is enhanced with additional inertial modality. Complementary visual and inertial modality are fused adaptively to provide accurate camera pose with a novel visual-inertial fusion Transformer, in which self-attention takes effect in visual-inertial feature interaction and cross-attention is utilized for task feature decoding and pose regression. Second, the monocular depth prior, which contains contextual information about the scene, is introduced to the multi-frame cost volume aggregation at the feature level. A novel monocular guided cost volume excitation module is proposed to adaptively modulate cost volume features and address possible matching ambiguity. With the proposed modules, a self-supervised multi-frame depth estimation network is presented, consisting of a monocular depth branch as prior, a camera pose branch integrating both visual and inertial modality, and a multi-frame depth branch producing the final depth with the aid of former two branches. Experimental results on the KITTI dataset show that our proposed method achieves notable performance boost on multi-frame depth estimation over the state-of-the-art competitors. Compared with ManyDepth and MOVEDepth, our method relatively improves depth accuracy by 9.2% and 5.3% on the KITTI dataset.

computer science, artificial intelligence, theory & methods
Monocular Depth Estimation Based on Dilated Convolutions and Feature Fusion

Hang Li,Shuai Liu,Bin Wang,Yuanhao Wu

DOI: https://doi.org/10.3390/app14135833

2024-01-01

Abstract:Depth estimation represents a prevalent research focus within the realm of computer vision. Existing depth estimation methodologies utilizing LiDAR (Light Detection and Ranging) technology typically obtain sparse depth data and are associated with elevated hardware expenses. Multi-view image-matching techniques necessitate prior knowledge of camera intrinsic parameters and frequently encounter challenges such as depth inconsistency, loss of details, and the blurring of edges. To tackle these challenges, the present study introduces a monocular depth estimation approach based on an end-to-end convolutional neural network. Specifically, a DNET backbone has been developed, incorporating dilated convolution and feature fusion mechanisms within the network architecture. By integrating semantic information from various receptive fields and levels, the model’s capacity for feature extraction is augmented, thereby enhancing its sensitivity to nuanced depth variations within the image. Furthermore, we introduce a loss function optimization algorithm specifically designed to address class imbalance, thereby enhancing the overall predictive accuracy of the model. Training and validation conducted on the NYU Depth-v2 (New York University Depth Dataset Version 2) and KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) datasets demonstrate that our approach outperforms other algorithms in terms of various evaluation metrics.
Depth Estimation from Multi-Scale SLIC Superpixels Using Non-Parametric Learning

Yifeng Jiang,Yuesheng Zhu,Yin Qing,Fan Yang

DOI: https://doi.org/10.1117/12.2281579

2017-01-01

Abstract:This study introduces a novel depth estimation method that can automatically generate plausible depth map from a single image with unstructured environment. Our goal is to extrapolate depth map with more correct, rich, and distinct depth order, which is both quantitatively accurate as well as visually pleasing. Based on the preexisting DepthTransfer algorithm, our approach primarily transfers depth information at the level of superpixels from the most photometrically similar retrieval images under the framework of non-parametric learning. Posteriorly, we propose to concurrently warp the corresponding superpixels in multi-scale levels, where we employ an improved SLIC technique to segment the RGBD images from coarse to fine. Then, modified Cross Bilateral Filter is leveraged to refine the final depth field. With respect to training and evaluation, we perform our experiment on the popular Make3D dataset and demonstrate that our method outperforms the state-of-the-art in both efficacy and computational efficiency. Especially, the final results show that in qualitatively evaluation, our results are visually superior in realism and simultaneously more immersive.
FusionDepth: Complement Self-Supervised Monocular Depth Estimation with Cost Volume

Zhuofei Huang,Jianlin Liu,Shang Xu,Ying Chen,Yong Liu

DOI: https://doi.org/10.48550/arXiv.2305.06036

2023-05-10

Computer Vision and Pattern Recognition

Abstract:Multi-view stereo depth estimation based on cost volume usually works better than self-supervised monocular depth estimation except for moving objects and low-textured surfaces. So in this paper, we propose a multi-frame depth estimation framework which monocular depth can be refined continuously by multi-frame sequential constraints, leveraging a Bayesian fusion layer within several iterations. Both monocular and multi-view networks can be trained with no depth supervision. Our method also enhances the interpretability when combining monocular estimation with multi-view cost volume. Detailed experiments show that our method surpasses state-of-the-art unsupervised methods utilizing single or multiple frames at test time on KITTI benchmark.
Unsupervised Learning-based Depth Estimation aided Visual SLAM Approach

Mingyang Geng,Suning Shang,Bo Ding,Huaimin Wang,Pengfei Zhang,Lei Zhang

DOI: https://doi.org/10.48550/arXiv.1901.07288

2019-01-22

Abstract:The RGB-D camera maintains a limited range for working and is hard to accurately measure the depth information in a far distance. Besides, the RGB-D camera will easily be influenced by strong lighting and other external factors, which will lead to a poor accuracy on the acquired environmental depth information. Recently, deep learning technologies have achieved great success in the visual SLAM area, which can directly learn high-level features from the visual inputs and improve the estimation accuracy of the depth information. Therefore, deep learning technologies maintain the potential to extend the source of the depth information and improve the performance of the SLAM system. However, the existing deep learning-based methods are mainly supervised and require a large amount of ground-truth depth data, which is hard to acquire because of the realistic constraints. In this paper, we first present an unsupervised learning framework, which not only uses image reconstruction for supervising but also exploits the pose estimation method to enhance the supervised signal and add training constraints for the task of monocular depth and camera motion estimation. Furthermore, we successfully exploit our unsupervised learning framework to assist the traditional ORB-SLAM system when the initialization module of ORB-SLAM method could not match enough features. Qualitative and quantitative experiments have shown that our unsupervised learning framework performs the depth estimation task comparable to the supervised methods and outperforms the previous state-of-the-art approach by $13.5\%$ on KITTI dataset. Besides, our unsupervised learning framework could significantly accelerate the initialization process of ORB-SLAM system and effectively improve the accuracy on environmental mapping in strong lighting and weak texture scenes.

Computer Vision and Pattern Recognition
FG-Depth: Flow-Guided Unsupervised Monocular Depth Estimation

Junyu Zhu,Lina Liu,Yong Liu,Wanlong Li,Feng Wen,Hongbo Zhang

DOI: https://doi.org/10.48550/arXiv.2301.08414

2023-02-07

Abstract:The great potential of unsupervised monocular depth estimation has been demonstrated by many works due to low annotation cost and impressive accuracy comparable to supervised methods. To further improve the performance, recent works mainly focus on designing more complex network structures and exploiting extra supervised information, e.g., semantic segmentation. These methods optimize the models by exploiting the reconstructed relationship between the target and reference images in varying degrees. However, previous methods prove that this image reconstruction optimization is prone to get trapped in local minima. In this paper, our core idea is to guide the optimization with prior knowledge from pretrained Flow-Net. And we show that the bottleneck of unsupervised monocular depth estimation can be broken with our simple but effective framework named FG-Depth. In particular, we propose (i) a flow distillation loss to replace the typical photometric loss that limits the capacity of the model and (ii) a prior flow based mask to remove invalid pixels that bring the noise in training loss. Extensive experiments demonstrate the effectiveness of each component, and our approach achieves state-of-the-art results on both KITTI and NYU-Depth-v2 datasets.

Computer Vision and Pattern Recognition,Artificial Intelligence
Unsupervised Monocular Estimation of Depth and Visual Odometry uUsing Attention and Depth-Pose Consistency Loss

Xiaogang Song,Haoyue Hu,Li Liang,Weiwei Shi,Guo Xie,Xiaofeng Lu,Xinhong Hei

DOI: https://doi.org/10.1109/tmm.2023.3312950

IF: 7.3

2023-01-01

IEEE Transactions on Multimedia

Abstract:Recent studies have shown that joint depth and pose estimation using convolutional neural networks (CNNs) can learn unlabelled monocular frames. However, three problems remain: 1) CNNs can only extract local features due to the limited receptive field, 2) scale ambiguity is inherent in the monocular task, and 3) illness regions violate the photometric consistency assumption and produce large errors. We propose a novel framework, ADPDepth, with corresponding effective strategies to ameliorate the above problems. First, a PCAtt module is designed to capture the correlation between channels and efficiently extract multiscale spatial information using a multibranch parallel strategy. Second, depth-pose consistency loss is proposed based on the geometric consistency in depth and pose to constrain the scale between samples, eliminate scale ambiguity and obtain a globally consistent scale. To further improve performance, a cover mask is derived from depth-pose consistency for filtering dynamic objects and outliers to reduce the adverse effects of these illness regions. Extensive experiments are conducted on the KITTI, NYU-Depth and Make3D datasets. Based on public benchmarks, the experimental results confirm that the proposed ADPDepth framework achieves state-of-the-art performance. The effectiveness of each strategy is also verified in subsequent ablation experiments.

computer science, information systems,telecommunications, software engineering
$\mathrm{F^2Depth}$: Self-supervised Indoor Monocular Depth Estimation via Optical Flow Consistency and Feature Map Synthesis

Xiaotong Guo,Huijie Zhao,Shuwei Shao,Xudong Li,Baochang Zhang

2024-03-27

Abstract:Self-supervised monocular depth estimation methods have been increasingly given much attention due to the benefit of not requiring large, labelled datasets. Such self-supervised methods require high-quality salient features and consequently suffer from severe performance drop for indoor scenes, where low-textured regions dominant in the scenes are almost indiscriminative. To address the issue, we propose a self-supervised indoor monocular depth estimation framework called $\mathrm{F^2Depth}$. A self-supervised optical flow estimation network is introduced to supervise depth learning. To improve optical flow estimation performance in low-textured areas, only some patches of points with more discriminative features are adopted for finetuning based on our well-designed patch-based photometric loss. The finetuned optical flow estimation network generates high-accuracy optical flow as a supervisory signal for depth estimation. Correspondingly, an optical flow consistency loss is designed. Multi-scale feature maps produced by finetuned optical flow estimation network perform warping to compute feature map synthesis loss as another supervisory signal for depth learning. Experimental results on the NYU Depth V2 dataset demonstrate the effectiveness of the framework and our proposed losses. To evaluate the generalization ability of our $\mathrm{F^2Depth}$, we collect a Campus Indoor depth dataset composed of approximately 1500 points selected from 99 images in 18 scenes. Zero-shot generalization experiments on 7-Scenes dataset and Campus Indoor achieve $\delta_1$ accuracy of 75.8% and 76.0% respectively. The accuracy results show that our model can generalize well to monocular images captured in unknown indoor scenes.

Computer Vision and Pattern Recognition
Monocular Depth Estimation With Affinity, Vertical Pooling, And Label Enhancement

Yukang Gan,Xiangyu Xu,Wenxiu Sun,Liang Lin

DOI: https://doi.org/10.1007/978-3-030-01219-9_14

2018-01-01

Abstract:Significant progress has been made in monocular depth estimation with Convolutional Neural Networks (CNNs). While absolute features, such as edges and textures, could be effectively extracted, the depth constraint of neighboring pixels, namely relative features, has been mostly ignored by recent CNN-based methods. To overcome this limitation, we explicitly model the relationships of different image locations with an affinity layer and combine absolute and relative features in an end-to-end network. In addition, we consider prior knowledge that major depth changes lie in the vertical direction, and thus, it is beneficial to capture long-range vertical features for refined depth estimation. In the proposed algorithm we introduce vertical pooling to aggregate image features vertically to improve the depth accuracy. Furthermore, since the Lidar depth ground truth is quite sparse, we enhance the depth labels by generating high-quality dense depth maps with off-the-shelf stereo matching method taking left-right image pairs as input. We also integrate multi-scale structure in our network to obtain global understanding of the image depth and exploit residual learning to help depth refinement. We demonstrate that the proposed algorithm performs favorably against state-of-the- methods both qualitatively and quantitatively on the KITTI driving dataset.

Unsupervised Learning Based Focal Stack Camera Depth Estimation

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Monocular Depth Estimation Based on Unsupervised Learning

Least Square Estimation Network for Depth Completion

Monocular Depth Estimation Based on Multi-Scale Graph Convolution Networks

FocDepthFormer: Transformer with latent LSTM for Depth Estimation from Focal Stack

FS-Depth: Focal-and-Scale Depth Estimation from a Single Image in Unseen Indoor Scene

Depth Estimation from Monocular Images Using Dilated Convolution and Uncertainty Learning.

Unsupervised Scale-Consistent Depth Learning from Video

Dynamic Fusion Network For Light Field Depth Estimation

Fully Self-Supervised Depth Estimation from Defocus Clue

Monocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference

Self-supervised multi-frame depth estimation with visual-inertial pose transformer and monocular guidance

Monocular Depth Estimation Based on Dilated Convolutions and Feature Fusion

Depth Estimation from Multi-Scale SLIC Superpixels Using Non-Parametric Learning

FusionDepth: Complement Self-Supervised Monocular Depth Estimation with Cost Volume

Unsupervised Learning-based Depth Estimation aided Visual SLAM Approach

FG-Depth: Flow-Guided Unsupervised Monocular Depth Estimation

Unsupervised Monocular Estimation of Depth and Visual Odometry uUsing Attention and Depth-Pose Consistency Loss

$\mathrm{F^2Depth}$: Self-supervised Indoor Monocular Depth Estimation via Optical Flow Consistency and Feature Map Synthesis

Monocular Depth Estimation With Affinity, Vertical Pooling, And Label Enhancement