Abstract:Numerous deep-learning methods have been successfully applied to semantic segmentation (SS) and height estimation (EH) of remote-sensing imagery. It has also been proved that such a framework can be reusable for multiple tasks to reduce computational resource overhead. However, there are still some technical limitations due to the semantic inconsistency between 3-D and 2-D features and the strong interference of different objects with similar spectral–spatial properties. Previous works have sought to address these issues through hard (HPS) or soft parameter sharing (SPS) schemes. But due to unintentional integration, the specific information transmitted between multiple tasks is not clear or in a lot of redundancy. Furthermore, tuning the weights by hand between classification and regression loss functions is challenging. In this article, a novel multitask learning (MTL) method, termed associatively segmenting semantics and estimating height (ASSEH), is proposed to associatively segment semantics and estimate height from monocular remote-sensing imagery. First, considering semantic inconsistency across tasks, we design a task-specific distillation (TSD) module containing a set of task-specific gating units (TSGUs) for each task at the cost of fewer parameters. The module allows for task-specific features to be tailored from the backbone while allowing for task-shared features to be transmitted. Second, we leverage the proposed cross-task propagation (CTP) module to construct and diffuse the local pattern graphlets at the common positions across tasks. Such a high-order recursive method can bridge two tasks explicitly to effectively settle semantic ambiguities caused by similar spectral characteristics with less computational burden and memory requirements. Third, a dynamic weighted geometric mean (DWGeoMean) strategy is introduced to dynamically learn the weights of each task and be more robust to the magnitude of the loss function. Finally, the results of the ISPRS Vaihingen dataset and the urban semantic 3-D (US3D) dataset well demonstrate that our ASSEH achieves state-of-the-art performance.

SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular Images

SDC-Depth: Semantic Divide-and-Conquer Network for Monocular Depth Estimation

Geometry-semantic Aware for Monocular 3D Semantic Scene Completion

SOSNet: Real-Time Small Object Segmentation via Hierarchical Decoding and Example Mining

Semantic Segmentation and Depth Estimation of Urban Road Scene Images Using Multi-Task Networks

SSNet: a joint learning network for semantic segmentation and disparity estimation

Depth-embedded Instance Segmentation Network for Urban Scene Parsing

S3Net: Innovating Stereo Matching and Semantic Segmentation with a Single-Branch Semantic Stereo Network in Satellite Epipolar Imagery

Hybridnet for depth estimation and semantic segmentation

CI-Net: a joint depth estimation and semantic segmentation network using contextual information

Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary

Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation

Surface-SOS: Self-Supervised Object Segmentation via Neural Surface Representation

JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds

Associatively Segmenting Semantics and Estimating Height from Monocular Remote-Sensing Imagery

DepthSSC: Depth-Spatial Alignment and Dynamic Voxel Resolution for Monocular 3D Semantic Scene Completion

Depth Estimation of Supervised Monocular Images Based on Semantic Segmentation.

Joint Unsupervised Learning of Depth, Pose, Ground Normal Vector and Ground Segmentation by a Monocular Camera Sensor.

Deep Convolutional Encoder–decoder Networks Based on Ensemble Learning for Semantic Segmentation of High-Resolution Aerial Imagery

A Fusion Network for Semantic Segmentation Using RGB-D Data

3SP-Net: Semantic Segmentation Network with Stereo Image Pairs for Urban Scene Parsing