Abstract:Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language models, we propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. However, unique challenges and potential domain gaps make this transfer not as easy as the first intuition. Firstly, a transformer-based tracker constructs unshared position embedding for template and search image. This poses a challenge for the transfer of LoRA, usually requiring consistency in the design when applied to the pre-trained backbone, to downstream tasks. Secondly, the inductive bias inherent in convolutional heads diminishes the effectiveness of parameter-efficient fine-tuning in tracking models. To overcome these limitations, we first decouple the position embeddings in transformer-based trackers into shared spatial ones and independent type ones. The shared embeddings, which describe the absolute coordinates of multi-resolution images (namely, the template and search images), are inherited from the pre-trained backbones. In contrast, the independent embeddings indicate the sources of each token and are learned from scratch. Furthermore, we design an anchor-free head solely based on MLP to adapt PETR, enabling better performance with less computational overhead. With our design, 1) it becomes practical to train trackers with the ViT-g backbone on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve the LaSOT SUC score from 0.703 to 0.742 with the L-224 variant; 4) we fast the inference speed of the L-224 variant from 52 to 119 FPS. Code and models are available at <a class="link-external link-https" href="https://github.com/LitingLin/LoRAT" rel="external noopener nofollow">this https URL</a>.

Robust Tracking via Unifying Pretrain-Finetuning and Visual Prompt Tuning

Visual tracking with screening region enrichment and target validation

PromptVT: Prompting for Efficient and Accurate Visual Tracking

Multi-features Guided Robust Visual Tracking.

OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning

Improving Visual Object Tracking through Visual Prompting

Visual Prompt Multi-Modal Tracking

High Performance Visual Object Tracking with Unified Convolutional Networks

Self-Prompting Tracking: A Fast and Efficient Tracking Pipeline for UAV Videos

PVT++: A Simple End-to-End Latency-Aware Visual Tracking Framework

UCT: Learning Unified Convolutional Networks for Real-time Visual Tracking

Adaptive Part Mining for Robust Visual Tracking.

Middle Fusion and Multi-Stage, Multi-Form Prompts for Robust RGB-T Tracking

PVP: Pre-trained Visual Parameter-Efficient Tuning

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

Motion-Driven Tracking via End-to-End Coarse-to-Fine Verifying

Refocus the Attention for Parameter-Efficient Thermal Infrared Object Tracking

VPPT: Visual Pre-Trained Prompt Tuning Framework for Few-Shot Image Classification

Hierarchical Tracking by Reinforcement Learning-Based Searching and Coarse-to-Fine Verifying

MixFormer: End-to-End Tracking With Iterative Mixed Attention

Unified Single-Stage Transformer Network for Efficient RGB-T Tracking