Abstract:Transformer-based methods have achieved remarkable success in various machine learning tasks. How to design efficient test-time adaptation methods for transformer models becomes an important research task. In this work, motivated by the dual-subband wavelet lifting scheme developed in multi-scale signal processing which is able to efficiently separate the input signals into principal components and noise components, we introduce a dual-path token lifting for domain shift correction in test time adaptation. Specifically, we introduce an extra token, referred to as \textit{domain shift token}, at each layer of the transformer network. We then perform dual-path lifting with interleaved token prediction and update between the path of domain shift tokens and the path of class tokens at all network layers. The prediction and update networks are learned in an adversarial manner. Specifically, the task of the prediction network is to learn the residual noise of domain shift which should be largely invariant across all classes and all samples in the target domain. In other words, the predicted domain shift noise should be indistinguishable between all sample classes. On the other hand, the task of the update network is to update the class tokens by removing the domain shift from the input image samples so that input samples become more discriminative between different classes in the feature space. To effectively learn the prediction and update networks with two adversarial tasks, both theoretically and practically, we demonstrate that it is necessary to use smooth optimization for the update network but non-smooth optimization for the prediction network. Experimental results on the benchmark datasets demonstrate that our proposed method significantly improves the online fully test-time domain adaptation performance. Code is available at \url{<a class="link-external link-https" href="https://github.com/yushuntang/DPAL" rel="external noopener nofollow">this https URL</a>}.

Dual-path Adaptation from Image to Video Transformers

TransVOS: Video Object Segmentation with Transformers

TP-VIT: A Two-Pathway Vision Transformer for Video Action Recognition

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Efficient Adaptation of Large Vision Transformer Via Adapter Re-Composing

Dynamic and Compressive Adaptation of Transformers From Images to Videos

Transferable-guided Attention Is All You Need for Video Domain Adaptation

Dual Path Transformer with Partition Attention

Dual stage black-box adversarial attack against vision transformer

Compressed Video Action Recognition with Dual-Stream and Dual-Modal Transformer

Time-, Memory- and Parameter-Efficient Visual Adaptation

Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation

Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers

Efficient Transfer Learning for Video-language Foundation Models

Adapting Short-Term Transformers for Action Detection in Untrimmed Videos

AIM: Adapting Image Models for Efficient Video Action Recognition

ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video

VideoTrack: Learning to Track Objects Via Video Transformer

A Dual-Feature-Based Adaptive Shared Transformer Network for Image Captioning

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias