Abstract:Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language models, we propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. However, unique challenges and potential domain gaps make this transfer not as easy as the first intuition. Firstly, a transformer-based tracker constructs unshared position embedding for template and search image. This poses a challenge for the transfer of LoRA, usually requiring consistency in the design when applied to the pre-trained backbone, to downstream tasks. Secondly, the inductive bias inherent in convolutional heads diminishes the effectiveness of parameter-efficient fine-tuning in tracking models. To overcome these limitations, we first decouple the position embeddings in transformer-based trackers into shared spatial ones and independent type ones. The shared embeddings, which describe the absolute coordinates of multi-resolution images (namely, the template and search images), are inherited from the pre-trained backbones. In contrast, the independent embeddings indicate the sources of each token and are learned from scratch. Furthermore, we design an anchor-free head solely based on MLP to adapt PETR, enabling better performance with less computational overhead. With our design, 1) it becomes practical to train trackers with the ViT-g backbone on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve the LaSOT SUC score from 0.703 to 0.742 with the L-224 variant; 4) we fast the inference speed of the L-224 variant from 52 to 119 FPS. Code and models are available at <a class="link-external link-https" href="https://github.com/LitingLin/LoRAT" rel="external noopener nofollow">this https URL</a>.

DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Grounding

VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control

ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts

Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

Towards a Unified View on Visual Parameter-Efficient Transfer Learning

Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition

Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization

A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models

Multi-Task Domain Adaptation for Language Grounding with 3D Objects

Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach

DoRA: Weight-Decomposed Low-Rank Adaptation

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension

ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

BarLeRIa: an Efficient Tuning Framework for Referring Image Segmentation.