Abstract:Semantic segmentation and disparity estimation are in the research frontier of the computer vision and remote sensing (RS) fields. However, existing methods mostly deal with these two problems separately or use a combination of multiple models to solve these two tasks. Due to a lack of sufficient information sharing and fusion, they still have difficulties in coping with seasonal appearance differences in 3-D RS problems. In this article, we propose a novel multitask learning architecture that considers the bottomx2013;up and upx2013;bottom visual attention mechanism for 3-D semantic detection, named bidirectional guided attention network (BGA-Net). BGA-Net consists of five modules: unified backbone module (UBM), bidirectional guided attention module (BGAM), semantic segmentation module (SSM), feature matching module (FMM), and bidirectional fusion module (BFM). First, in UBM, we use a shared backbone to extract unified features and share them with three branches/modules (BGAM, SSM, and FMM). Then, SSM and FMM branches are applied to estimate segmentation and disparity maps, whereas the third branch/module (BGAM) shares the global features to guide the task-specific learning via attention mechanism. Finally, we fuse the results of the two tasks by BFM to improve the final performance. Extensive experiments demonstrate that: 1) our BGA-Net can handle the two tasks simultaneously and can be trained in an end-to-end way; 2) these modules fully take advantage of the two tasksx2019; information to share features and enhance the scene understanding ability, effectively against seasons change of RS images; and 3) BGA-Net has notable superiority and greater flexibility and also sets a new state of the art on the urban semantic 3-D (US3D) benchmark. Moreover, BGA-Net also provides insights into the intelligent interpretation of RS data images.

SDBF-Net: Semantic and Disparity Bidirectional Fusion Network for 3D Semantic Detection on Incidental Satellite Images.

Attention-based Multi-modal Fusion Network for Semantic Scene Completion.

ObjectFusion: an Object Detection and Segmentation Framework with RGB-D SLAM and Convolutional Neural Networks

S3Net: Innovating Stereo Matching and Semantic Segmentation with a Single-Branch Semantic Stereo Network in Satellite Epipolar Imagery

SemStereo: Semantic-Constrained Stereo Matching Network for Remote Sensing

Bidirectional Guided Attention Network for 3-D Semantic Detection of Remote Sensing Images

Semantic Stereo for Incidental Satellite Images

S^3M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving

BF3D: Bi-directional Fusion 3D Detector with Semantic Sampling and Geometric Mapping

Semantic 3d Reconstruction Using Multi-View High-Resolution Satellite Images Based On U-Net And Image-Guided Depth Fusion

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

S$^3$M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving

Semantic Consistency Networks for 3D Object Detection.

Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection

MSDC-Net: Multi-Scale Dense and Contextual Networks for Automated Disparity Map for Stereo Matching

RGB and LiDAR Fusion-based 3D Semantic Segmentation for Autonomous Driving

RGB-D Semantic Segmentation and Label-Oriented Voxelgrid Fusion for Accurate 3D Semantic Mapping

Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion

A Multi-Level Semantic Fusion VoteNet for 3D Object Detection on Point Clouds

Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentation

MSDC-Net: Multi-Scale Dense and Contextual Networks for Stereo Matching.