Abstract:Intelligent vehicles clearly benefit from the expanded Field of View (FoV) of the 360-degree sensors, but the vast majority of available semantic segmentation training images are captured with pinhole cameras. In this work, we look at this problem through the lens of domain adaptation and bring panoramic semantic segmentation to a setting, where labelled training data originates from a different distribution of conventional pinhole camera images. First, we formalize the task of unsupervised domain adaptation for panoramic semantic segmentation, where a network trained on labelled examples from the source domain of pinhole camera data is deployed in a different target domain of panoramic images, for which no labels are available. To validate this idea, we collect and publicly release DensePASS - a novel densely annotated dataset for panoramic segmentation under cross-domain conditions, specifically built to study the Pinhole-to-Panoramic transfer and accompanied with pinhole camera training examples obtained from Cityscapes. DensePASS covers both, labelled- and unlabelled 360-degree images, with the labelled data comprising 19 classes which explicitly fit the categories available in the source domain (i.e. pinhole) data. To meet the challenge of domain shift, we leverage the current progress of attention-based mechanisms and build a generic framework for cross-domain panoramic semantic segmentation based on different variants of attention-augmented domain adaptation modules. Our framework facilitates information exchange at local- and global levels when learning the domain correspondences and improves the domain adaptation performance of two standard segmentation networks by 6.05% and 11.26% in Mean IoU.

What problem does this paper attempt to address?

The paper attempts to address the challenge of domain adaptation in semantic segmentation on panoramic images. Specifically: 1. **Data Distribution Discrepancy**: Most existing semantic segmentation training datasets are based on images captured by pinhole cameras, while datasets for panoramic images are relatively scarce and costly to annotate. Therefore, directly applying models trained on pinhole camera images to panoramic images results in significant performance degradation. 2. **Geometric Distortion**: Panoramic images undergo equirectangular projection, which causes horizontal stretching and geometric distortion, making them significantly different from standard pinhole camera images and leading to a substantial domain gap. To address these issues, the authors propose a new benchmark dataset named DensePASS and develop an unsupervised domain adaptation framework (P2PDA) that improves adaptation performance from the pinhole camera domain to the panoramic image domain through attention mechanisms and region context exchange. The specific contributions include: - Creating and publicly releasing the DensePASS dataset, which contains panoramic images from around the world annotated with 19 categories. - Formalizing the task of unsupervised domain adaptation in panoramic semantic segmentation, focusing on transferring from label-rich pinhole camera datasets to label-scarce panoramic image datasets. - Proposing a general P2PDA framework and investigating different domain adaptation modules, including the Segmentation Domain Adaptation Module (SDAM), Attention Domain Adaptation Module (ADAM), and Region Context Domain Adaptation Module (RCDAM). Through these methods, the authors aim to improve the performance of semantic segmentation on panoramic images and promote research on panoramic semantic segmentation under cross-domain conditions.

DensePASS: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange

Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation

PASS: Panoramic Annular Semantic Segmentation

DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation Through SwaftNet for Surrounding Sensing

Can We PASS Beyond the Field of View? Panoramic Annular Semantic Segmentation for Real-World Surrounding Perception

Aerial-PASS: Panoramic Annular Scene Segmentation in Drone Videos

Multi-source Domain Adaptation for Panoramic Semantic Segmentation

Omnisupervised Omnidirectional Semantic Segmentation

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

ADeLA: Automatic Dense Labeling with Attention for Viewpoint Shift in Semantic Segmentation

When Masked Image Modeling Meets Source-free Unsupervised Domain Adaptation: Dual-Level Masked Network for Semantic Segmentation

Pass3d: Precise And Accelerated Semantic Segmentation For 3d Point Cloud

Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation

EDAPS: Enhanced Domain-Adaptive Panoptic Segmentation

Both Style and Distortion Matter: Dual-Path Unsupervised Domain Adaptation for Panoramic Semantic Segmentation

Occlusion-Aware Seamless Segmentation

Look at the Neighbor: Distortion-aware Unsupervised Domain Adaptation for Panoramic Semantic Segmentation

360SFUDA++: Towards Source-free UDA for Panoramic Segmentation by Learning Reliable Category Prototypes

SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation

Capturing Omni-Range Context for Omnidirectional Segmentation

Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning