Dynamic Domains, Dynamic Solutions: DPCore for Continual Test-Time Adaptation

Yunbei Zhang,Akshay Mehra,Jihun Hamm

2024-08-26

Abstract:Continual Test-Time Adaptation (CTTA) seeks to adapt a source pre-trained model to continually changing, unlabeled target domains. Existing TTA methods are typically designed for environments where domain changes occur sequentially and can struggle in more dynamic scenarios, as illustrated in Figure \ref{fig:settings}. Inspired by the principles of online K-Means, we introduce a novel approach to CTTA through visual prompting. We propose a \emph{Dynamic Prompt Coreset} that not only preserves knowledge from previously visited domains but also accommodates learning from new potential domains. This is complemented by a distance-based \emph{Weight Updating Mechanism} that ensures the coreset remains current and relevant. Our approach employs a fixed model architecture alongside the coreset and an innovative updating system to effectively mitigate challenges such as catastrophic forgetting and error accumulation. Extensive testing on four widely-used benchmarks demonstrates that our method consistently outperforms state-of-the-art alternatives in both classification and segmentation CTTA tasks across the structured and dynamic CTTA settings, with $99\%$ fewer trainable parameters.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the issue of Continual Test-Time Adaptation (CTTA) in a continuously changing and unlabeled target domain. Specifically:

1. **Adaptability in Dynamically Changing Environments**:
- Existing CTTA methods usually assume that changes in the target domain are sequential and predictable (Continual Structured Change, CSC), but in real-world applications, domain changes are often rapid and unpredictable (Continual Dynamic Change, CDC).
- To cope with such dynamically changing environments, the paper proposes a novel method called DPCore.

2. **Avoiding Catastrophic Forgetting and Error Accumulation**:
- In a continuously changing environment, the model needs to effectively adapt to new potential domains without forgetting previously learned knowledge.
- DPCore achieves this goal through a Dynamic Prompt Coreset and a distance-based weight update mechanism.

3. **Efficient and Lightweight Solution**:
- DPCore not only performs well on classification tasks but also outperforms existing methods in semantic segmentation tasks, while significantly reducing the number of parameters used, thereby improving efficiency in practical applications.

### Main Contributions
- Proposes a new method for CTTA called DPCore, which uses a Dynamic Prompt Coreset and a weighted update mechanism to store and update the model's experience, effectively addressing challenges in continual learning such as catastrophic forgetting and error accumulation.
- Introduces a more realistic and challenging CTTA setting—Continual Dynamic Change (CDC), where domain changes are rapid and unpredictable, with varying durations and timings.
- Validates the effectiveness and efficiency of DPCore on multiple widely-used benchmark datasets and provides an analysis of its robustness to domain changes, demonstrating its superior adaptability.