AWF: Adaptive Weight Fusion for Enhanced Class Incremental Semantic Segmentation

Zechao Sun,Haolin Jin,Weitong Chen,Luping Zhou

2024-09-13

Abstract:Class Incremental Semantic Segmentation (CISS) aims to mitigate catastrophic forgetting by maintaining a balance between previously learned and newly introduced knowledge. Existing methods, primarily based on regularization techniques like knowledge distillation, help preserve old knowledge but often face challenges in effectively integrating new knowledge, resulting in limited overall improvement. Endpoints Weight Fusion (EWF) method, while simple, effectively addresses some of these limitations by dynamically fusing the model weights from previous steps with those from the current step, using a fusion parameter alpha determined by the relative number of previously known classes and newly introduced classes. However, the simplicity of the alpha calculation may limit its ability to fully capture the complexities of different task scenarios, potentially leading to suboptimal fusion outcomes. In this paper, we propose an enhanced approach called Adaptive Weight Fusion (AWF), which introduces an alternating training strategy for the fusion parameter, allowing for more flexible and adaptive weight integration. AWF achieves superior performance by better balancing the retention of old knowledge with the learning of new classes, significantly improving results on benchmark CISS tasks compared to the original EWF. And our experiment code will be released on Github.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue of how to better balance the retention of old knowledge and the learning of new knowledge in Class Incremental Semantic Segmentation (CISS) to mitigate catastrophic forgetting. Specifically:

1. **Limitations of Existing Methods**: Most existing CISS methods are based on regularization techniques (such as knowledge distillation). While these methods help retain old knowledge, they face challenges in effectively integrating new knowledge, leading to limited overall performance improvement.

2. **Shortcomings of the EWF Method**: Although the Endpoints Weight Fusion (EWF) method alleviates some issues by dynamically fusing model weights, its fixed fusion parameter α may not adequately capture the relationship between old and new knowledge in complex scenarios, resulting in suboptimal outcomes.

3. **Proposed New Method**: The paper proposes an improved method—Adaptive Weight Fusion (AWF), which introduces a trainable fusion parameter α. This parameter is optimized through an alternating training strategy, allowing the model to more flexibly adapt to the data characteristics of different tasks, thereby effectively learning new categories while retaining old knowledge.

With these improvements, AWF significantly enhances performance compared to EWF on benchmark CISS tasks, particularly excelling in scenarios where a large number of categories need to be added at each incremental step.