AViTMP: A Tracking-Specific Transformer for Single-Branch Visual Tracking

Chuanming Tang,Kai Wang,Joost van de Weijer,Jianlin Zhang,Yongmei Huang

DOI: https://doi.org/10.1109/TIV.2024.3422806

2024-07-04

Abstract:Visual object tracking is a fundamental component of transportation systems, especially for intelligent driving. Despite achieving state-of-the-art performance in visual tracking, recent single-branch trackers tend to overlook the weak prior assumptions associated with the Vision Transformer (ViT) encoder and inference pipeline in visual tracking. Moreover, the effectiveness of discriminative trackers remains constrained due to the adoption of the dual-branch pipeline. To tackle the inferior effectiveness of vanilla ViT, we propose an Adaptive ViT Model Prediction tracker (AViTMP) to design a customised tracking method. This method bridges the single-branch network with discriminative models for the first time. Specifically, in the proposed encoder AViT encoder, we introduce a tracking-tailored Adaptor module for vanilla ViT and a joint target state embedding to enrich the target-prior embedding paradigm. Then, we combine the AViT encoder with a discriminative transformer-specific model predictor to predict the accurate location. Furthermore, to mitigate the limitations of conventional inference practice, we present a novel inference pipeline called CycleTrack, which bolsters the tracking robustness in the presence of distractors via bidirectional cycle tracking verification. In the experiments, we evaluated AViTMP on eight tracking benchmarks for a comprehensive assessment, including LaSOT, LaSOTExtSub, AVisT, etc. The experimental results unequivocally establish that, under fair comparison, AViTMP achieves state-of-the-art performance, especially in terms of long-term tracking and robustness. The source code will be released at <a class="link-external link-https" href="https://github.com/Tchuanm/AViTMP" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper primarily addresses the challenges of Visual Object Tracking (VOT) in intelligent driving systems by proposing an improved single-branch tracking method. Specifically, the paper attempts to solve the following key issues:

1. **Limitations of Vision Transformer (ViT) in Visual Tracking**:
- Current single-branch trackers tend to overlook the inherent weak prior assumptions in the ViT encoder and inference pipeline, leading to slower convergence and suboptimal performance.

2. **Limited Effectiveness of Dual-Branch Trackers**:
- Dual-branch trackers have limited discriminative ability due to their dual-branch pipeline.

3. **Lack of Image-Specific Inductive Bias**:
- The original ViT architecture lacks image-specific inductive bias, which limits its performance in visual tracking tasks.

To address the above issues, the authors propose the Adaptive ViT Model Prediction Tracker (A ViTMP), which includes the following core components:

- **Adaptive ViT Encoder (A ViT-Encoder)**:
- Introduces an Adaptor module and Joint State Embedding tailored for tracking tasks to enhance target prior information.

- **Dense-Fusion Decoder**:
- Generates model weights through a dense-fusion decoder and combines them with encoded features for target prediction.

Additionally, the paper proposes a new inference mechanism—CycleTrack—to improve robustness and consistency in long-term tracking.

Through experimental evaluations on multiple benchmark datasets, A ViTMP demonstrates excellent performance in long-term tracking and robustness, achieving state-of-the-art levels.