Abstract:Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the real-world. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering the model incapable of focusing only on relevant complementary information for fusion. To address this limitation, we propose a mutimodal semantic segmentation framework that dynamically adapts the fusion of modality-specific features while being sensitive to the object category, spatial location and scene context in a self-supervised manner. Specifically, we propose an architecture consisting of two modality-specific encoder streams that fuse intermediate encoder representations into a single decoder using our proposed self-supervised model adaptation fusion mechanism which optimally combines complementary features. As intermediate representations are not aligned across modalities, we introduce an attention scheme for better correlation. In addition, we propose a computationally efficient unimodal segmentation architecture termed AdapNet++ that incorporates a new encoder with multiscale residual units and an efficient atrous spatial pyramid pooling that has a larger effective receptive field with more than 10x fewer parameters, complemented with a strong decoder with a multi-resolution supervision scheme that recovers high-resolution details. Comprehensive empirical evaluations on several benchmarks demonstrate that both our unimodal and multimodal architectures achieve state-of-the-art performance.

FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation.

When Masked Image Modeling Meets Source-free Unsupervised Domain Adaptation: Dual-Level Masked Network for Semantic Segmentation

Joint Feature-Level And Pixel-Level Domain Adaption For Object Detection In The Wild

ADeLA: Automatic Dense Labeling with Attention for Viewpoint Shift in Semantic Segmentation

Fully Convolutional Adaptation Networks for Semantic Segmentation

Pixel-Level Domain Adaptation: A New Perspective for Enhancing Weakly Supervised Semantic Segmentation

CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Gotta Adapt 'Em All: Joint Pixel and Feature-Level Domain Adaptation for Recognition in the Wild

Unsupervised Domain Adaption for High-Resolution Coastal Land Cover Mapping with Category-Space Constrained Adversarial Network

Taking a Closer Look at Domain Shift: Category-Level Adversaries for Semantics Consistent Domain Adaptation

Constrained Domain Adaptation for Image Segmentation

Classification Constrained Discriminator for Domain Adaptive Semantic Segmentation

Knowledge Adaptation for Efficient Semantic Segmentation

Affinity Space Adaptation for Semantic Segmentation Across Domains

ACE: Adapting to Changing Environments for Semantic Segmentation

Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning

Domain Adaptation for Remote Sensing Image Semantic Segmentation: An Integrated Approach of Contrastive Learning and Adversarial Learning

Class-conditional domain adaptation for semantic segmentation

Classes Matter: A Fine-grained Adversarial Approach to Cross-domain Semantic Segmentation

Self-Supervised Model Adaptation for Multimodal Semantic Segmentation

Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation