Abstract:Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language models, we propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. However, unique challenges and potential domain gaps make this transfer not as easy as the first intuition. Firstly, a transformer-based tracker constructs unshared position embedding for template and search image. This poses a challenge for the transfer of LoRA, usually requiring consistency in the design when applied to the pre-trained backbone, to downstream tasks. Secondly, the inductive bias inherent in convolutional heads diminishes the effectiveness of parameter-efficient fine-tuning in tracking models. To overcome these limitations, we first decouple the position embeddings in transformer-based trackers into shared spatial ones and independent type ones. The shared embeddings, which describe the absolute coordinates of multi-resolution images (namely, the template and search images), are inherited from the pre-trained backbones. In contrast, the independent embeddings indicate the sources of each token and are learned from scratch. Furthermore, we design an anchor-free head solely based on MLP to adapt PETR, enabling better performance with less computational overhead. With our design, 1) it becomes practical to train trackers with the ViT-g backbone on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve the LaSOT SUC score from 0.703 to 0.742 with the L-224 variant; 4) we fast the inference speed of the L-224 variant from 52 to 119 FPS. Code and models are available at <a class="link-external link-https" href="https://github.com/LitingLin/LoRAT" rel="external noopener nofollow">this https URL</a>.

Refocus the Attention for Parameter-Efficient Thermal Infrared Object Tracking

A Fourier-Transform-Based Framework with Asymptotic Attention for Mobile Thermal InfraRed Object Detection

MRF3Net: Infrared Small Target Detection Using Multi-Receptive Field Perception and Effective Feature Fusion

Real-Time RGBT Target Tracking Based on Attention Mechanism

Learning Dual-Level Deep Representation for Thermal Infrared Tracking

Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation

Efficient thermal infrared tracking with cross-modal compress distillation

RCFT: re-parameterization convolution and feature filter for object tracking

Unsupervised RGB-T object tracking with attentional multi-modal feature fusion

Feature enhancement and coarse-to-fine detection for RGB-D tracking

RGB-T Tracking Based on Mixed Attention

Exploring reliable infrared object tracking with spatio-temporal fusion transformer

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

RGB-T tracking by modality difference reduction and feature re-selection

OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning

RGB-T Tracking with Template-Bridged Search Interaction and Target-Preserved Template Updating

Progressive Domain Adaptation for Thermal Infrared Object Tracking

In defense and revival of Bayesian filtering for thermal infrared object tracking

Unsupervised Cross-Modal Distillation for Thermal Infrared Tracking

Asymmetric Deformable Spatio-temporal Framework for Infrared Object Tracking

Attention fusion and target-uncertain detection for discriminative tracking