Cs2K: Class-specific and Class-shared Knowledge Guidance for Incremental Semantic Segmentation

Wei Cong,Yang Cong,Yuyang Liu,Gan Sun

2024-07-12

Abstract:Incremental semantic segmentation endeavors to segment newly encountered classes while maintaining knowledge of old classes. However, existing methods either 1) lack guidance from class-specific knowledge (i.e., old class prototypes), leading to a bias towards new classes, or 2) constrain class-shared knowledge (i.e., old model weights) excessively without discrimination, resulting in a preference for old classes. In this paper, to trade off model performance, we propose the Class-specific and Class-shared Knowledge (Cs2K) guidance for incremental semantic segmentation. Specifically, from the class-specific knowledge aspect, we design a prototype-guided pseudo labeling that exploits feature proximity from prototypes to correct pseudo labels, thereby overcoming catastrophic forgetting. Meanwhile, we develop a prototype-guided class adaptation that aligns class distribution across datasets via learning old augmented prototypes. Moreover, from the class-shared knowledge aspect, we propose a weight-guided selective consolidation to strengthen old memory while maintaining new memory by integrating old and new model weights based on weight importance relative to old classes. Experiments on public datasets demonstrate that our proposed Cs2K significantly improves segmentation performance and is plug-and-play.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in Incremental Semantic Segmentation (ISS), how to maintain the memory of old classes while effectively identifying new classes when introducing new classes. Existing methods either lack the guidance of class - specific knowledge from old - class prototypes, resulting in a bias towards new classes; or overly restrict class - shared knowledge (i.e., old model weights), resulting in a bias towards old classes. These problems will lead to catastrophic forgetting, that is, the phenomenon that the model forgets old data when learning new data.

To solve the above problems, the paper proposes the Class - specific and Class - shared Knowledge (Cs2K) - guided model, aiming to balance the model performance and achieve it in the following ways:

1. **In terms of class - specific knowledge**:
- **Prototype - guided pseudo - labels**: Use the proximity between features and prototypes to correct pseudo - labels, thereby overcoming catastrophic forgetting.
- **Prototype - guided class adaptation**: Align the class distributions between different datasets by learning enhanced old - class prototypes.

2. **In terms of class - shared knowledge**:
- **Weight - guided selective consolidation**: Selectively integrate the weights of the old model and the new model based on the importance of weights for old classes, to strengthen old memories while maintaining new memories.

Through these methods, the experimental results of the Cs2K model on multiple public datasets show that it significantly improves the segmentation performance and has good plug - in application characteristics.