Abstract:Continual test-time adaptation (CTTA) has recently emerged to adapt a pre-trained source model to continuously evolving target distributions, which accommodates the dynamic nature of real-world environments. To mitigate the risk of catastrophic forgetting in CTTA, existing methods typically incorporate explicit regularization terms to constrain the variation of model parameters. However, they cannot fundamentally resolve catastrophic forgetting because they rely on a single shared model to adapt across all target domains, which inevitably leads to severe inter-domain interference. In this paper, we introduce learnable domain-specific prompts that guide the model to adapt to corresponding target domains, thereby partially disentangling the parameter space of different domains. In the absence of domain identity for target samples, we propose a novel dynamic Prompt AllocatIon aNd Tuning (PAINT) method, which utilizes a query mechanism to dynamically determine whether the current samples come from a known domain or an unexplored one. For known domains, the corresponding domain-specific prompt is directly selected, while for previously unseen domains, a new prompt is allocated. Prompt tuning is subsequently performed using mutual information maximization along with structural regularization. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our PAINT method for CTTA. We have released our code at <a class="link-external link-https" href="https://github.com/Cadezzyr/PAINT" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of **catastrophic forgetting** in **Continual Test - Time Adaptation (CTTA)**. Specifically, CTTA refers to making a pre - trained model be able to adapt to the constantly changing target distributions to deal with the dynamically changing environments in the real world. However, in this process, the model may forget the knowledge it has learned before, which is so - called catastrophic forgetting. #### Reasons for Catastrophic Forgetting Existing CTTA methods usually rely on a single shared model to adapt to all target domains, which will lead to severe **inter - domain interference**. When the shared model adapts to a new target domain, its parameters will be updated to optimize the performance on new data, but this process may overwrite the parameters that were crucial for the old domains before, thus leading to catastrophic forgetting. #### Solutions To solve this problem, the authors propose a new method named **PAINT (Prompt AllocatIon aNd Tuning)**. PAINT guides the model to adapt to different target domains by introducing **domain - specific prompts**, thus partially decoupling the parameter spaces of different domains. Specifically, PAINT includes the following key steps: 1. **Dynamic Prompt Allocation**: - Use a query mechanism to dynamically determine whether the current sample is from a known domain or an unexplored domain. - For a known domain, select the corresponding domain - specific prompt; for an unseen domain, allocate a new prompt. 2. **Prompt Tuning**: - Use **mutual information maximization** and **structural regularization** to tune the prompt. - Mutual information maximization can improve the individual certainty of prediction and global diversity, reducing the situation where the model prediction collapses into a few categories. - Structural regularization ensures the consistency between the prediction of the interpolated target sample and the label of the interpolated sample, enhancing the model's generalization ability on the target domain. 3. **Prompt Key Updating**: - Use a moving - average strategy to update the prompt key, making it gradually stable and aligned with the target domain. Through these methods, PAINT can effectively adapt to the constantly changing target distributions without relying on additional storage and significantly reduce the risk of catastrophic forgetting. #### Experimental Results Experiments show that PAINT has achieved state - of - the - art performance on three benchmark datasets (CIFAR10 - C, ImageNet - C and ImageNet - R), verifying its effectiveness and superiority. ### Summary The core problem of this paper is to solve the catastrophic forgetting problem in CTTA. By introducing domain - specific prompts and a dynamic allocation mechanism, PAINT can partially decouple the parameter spaces of different domains, thus effectively alleviating this problem.

Dynamic Prompt Allocation and Tuning for Continual Test-Time Adaptation

Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation

Dynamic Domains, Dynamic Solutions: DPCore for Continual Test-Time Adaptation

Parameter-Selective Continual Test-Time Adaptation

Exploring Test-Time Adaptation for Object Detection in Continually Changing Environments

A Versatile Framework for Continual Test-Time Domain Adaptation: Balancing Discriminability and Generalizability

Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation

Historical Test-time Prompt Tuning for Vision Foundation Models

PALM: Pushing Adaptive Learning Rate Mechanisms for Continual Test-Time Adaptation

DART: Dual-Modal Adaptive Online Prompting and Knowledge Retention for Test-Time Adaptation

ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation

Robust Test-Time Adaptation for Zero-Shot Prompt Tuning

Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation

Continual Test-time Domain Adaptation via Dynamic Sample Selection

Dynamic Prompting: A Unified Framework for Prompt Tuning

Decoupled Prompt-Adapter Tuning for Continual Activity Recognition

C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion

Maintain Plasticity in Long-timescale Continual Test-time Adaptation

Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation

Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning

Diffusion-Enhanced Test-time Adaptation with Text and Image Augmentation