Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

Zhitong Gao,Bingnan Li,Mathieu Salzmann,Xuming He
2024-11-06
Abstract:In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor out-of-distribution (OOD) detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at <a class="link-external link-https" href="https://github.com/gaozhitong/MultiShiftSeg" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### The Problem the Paper Attempts to Solve In open-world scenarios, novel categories and new domains may coexist. An ideal segmentation model should be able to detect anomalous categories to ensure safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, resulting in poor performance in Out-of-Distribution (OOD) detection or domain generalization. This paper aims to enable the model to effectively generalize in regions of covariate shift while accurately identifying regions of semantic shift. To achieve this goal, the authors designed a novel generative augmentation method that produces coherent images containing anomalous (or novel) objects and various covariate shifts. They also introduced a training strategy that recalibrates uncertainty specific to semantic shifts and enhances the feature extractor to align features related to domain shifts. Through these methods, the authors validated their approach on multiple benchmarks, including semantic and domain shifts. Experimental results show that this method achieves state-of-the-art performance across all benchmarks, improving both OOD detection capability and domain generalization performance.