Abstract:Estimating uncertainty for deep networks is a good way to assess the reliability of prediction results and helps self-driving cars avoid traffic accidents. Compared to single RGB or thermal image segmentation, fusing RGB and thermal adds more sources of uncertainty to object boundaries including blurred boundaries of thermal images and the discrepancy of object boundary between RGB and thermal images because the objects can not be perfectly aligned. Besides, it is important to model the relationship between uncertainty and segmentation quality of RGB-T segmentation. To address these challenges, we propose Region of Interest uncertainty (RoI uncertainty) to focus on the uncertainty inside an object with indicative meaning and ignore the boundary uncertainty due to the imaging mechanism and alignment mechanism. We improve the correlation between accuracy and RoI uncertainty compared to the correlation between accuracy and conventional uncertainty. The RoI uncertainty can be obtained not only by morphological operations based on estimated uncertainty but by deep learning-based RoI uncertainty networks using uncertainty as input. We design two types of RoI uncertainty estimation networks, one using RoI as labels called RoI-Net and the other using region uncertainty as labels called RoI-Unc-Net. Furthermore, we compare three uncertainty estimation methods including Monte Carlo Dropout (MC Dropout), Deep Ensemble, and MC Dropout Ensemble, and three measures including predictive entropy, mutual information, and variance that are introduced to quantify uncertainty. From extensive experiments on the RGB-T dataset, MC Dropout with entropy measure shows better uncertainty estimation and model calibration. The RoI uncertainty based on MC Dropout with entropy can be used to show the segmentation quality. Besides, it is an unsupervised metric to indicate the degree of risk of the segmentation result in real road scenes. We compare the growth rate of RoI uncertainty on several state-of-the-art RGB-T fusion methods, and the results show that fusing RGB and thermal features in multiple stages is more effective than a single-stage fusion strategy.

Uncertainty-aware RGBD Image Segmentation.

Rough-fuzzy set approach for color and texture based image segmentation

ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation.

RGB-D Video Segmentation via Geodesic Spatio-Temporal Propagation

RGB-D image segmentation using superpixel and multi-feature fusion graph theory

Uncertainty-Aware Self-Improving Framework for Depth Estimation

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

Depth Estimation from Monocular Images Using Dilated Convolution and Uncertainty Learning.

Probabilistic RGB-D odometry based on points, lines and planes under depth uncertainty

Depth Segmentation in Real-World Scenes Based on U-V Disparity Analysis

RoI Uncertainty for RGB-Thermal Image Segmentation.

Spatio-Temporal Segmentation with Depth-Inferred Videos of Static Scenes

RGB×D: Learning Depth-Weighted RGB Patches for RGB-D Indoor Semantic Segmentation

Depth-aware CNN for RGB-D Segmentation

Weakly-Supervised RGBD Video Object Segmentation

Joining Geometric and RGB Features for RGB-D Semantic Segmentation

Interactive RGB-D Image Segmentation Using Hierarchical Graph Cut and Geodesic Distance

Spatial Uncertainty Model Based on Scale-Space for RGBD-SLAM System

Hierarchical Uncertainty Estimation for Medical Image Segmentation Networks

RGB-Depth Structure Similarity for Self-supervised Monocular Depth Estimation

Spatial Uncertainty Model of a Three-View RGB-D Camera System