Abstract:Numerous deep-learning methods have been successfully applied to semantic segmentation (SS) and height estimation (EH) of remote-sensing imagery. It has also been proved that such a framework can be reusable for multiple tasks to reduce computational resource overhead. However, there are still some technical limitations due to the semantic inconsistency between 3-D and 2-D features and the strong interference of different objects with similar spectral–spatial properties. Previous works have sought to address these issues through hard (HPS) or soft parameter sharing (SPS) schemes. But due to unintentional integration, the specific information transmitted between multiple tasks is not clear or in a lot of redundancy. Furthermore, tuning the weights by hand between classification and regression loss functions is challenging. In this article, a novel multitask learning (MTL) method, termed associatively segmenting semantics and estimating height (ASSEH), is proposed to associatively segment semantics and estimate height from monocular remote-sensing imagery. First, considering semantic inconsistency across tasks, we design a task-specific distillation (TSD) module containing a set of task-specific gating units (TSGUs) for each task at the cost of fewer parameters. The module allows for task-specific features to be tailored from the backbone while allowing for task-shared features to be transmitted. Second, we leverage the proposed cross-task propagation (CTP) module to construct and diffuse the local pattern graphlets at the common positions across tasks. Such a high-order recursive method can bridge two tasks explicitly to effectively settle semantic ambiguities caused by similar spectral characteristics with less computational burden and memory requirements. Third, a dynamic weighted geometric mean (DWGeoMean) strategy is introduced to dynamically learn the weights of each task and be more robust to the magnitude of the loss function. Finally, the results of the ISPRS Vaihingen dataset and the urban semantic 3-D (US3D) dataset well demonstrate that our ASSEH achieves state-of-the-art performance.

Joint Task-Recursive Learning for Semantic Segmentation and Depth Estimation

Joint Task-Recursive Learning for RGB-D Scene Understanding

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion

Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary

Joint Learning of Semantic Segmentation and Height Estimation for Remote Sensing Image Leveraging Contrastive Learning

Boosting Unsupervised Monocular Depth Estimation with Auxiliary Semantic Information

Joint-Confidence-Guided Multi-Task Learning for 3D Reconstruction and Understanding from Monocular Camera

Joint Object Segmentation and Depth Upsampling

SDC-Depth: Semantic Divide-and-Conquer Network for Monocular Depth Estimation

SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular Images

Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation

Semantic Segmentation and Depth Estimation Based on Residual Attention Mechanism

TRL: Transformer Based Refinement Learning for Hybrid-Supervised Semantic Segmentation

Semantic Segmentation and Depth Estimation of Urban Road Scene Images Using Multi-Task Networks

SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images

SemHint-MD: Learning from Noisy Semantic Labels for Self-Supervised Monocular Depth Estimation

Associatively Segmenting Semantics and Estimating Height from Monocular Remote-Sensing Imagery

Collaborative Learning of Depth Estimation, Visual Odometry and Camera Relocalization from Monocular Videos.

Learning Monocular Depth in Dynamic Environment via Context-aware Temporal Attention