Abstract:Recently, some researchers have begun to adopt the Transformer to combine or replace the widely used ResNet as their new backbone network. As the Transformer captures the long-range relations between pixels well using the self-attention scheme, which complements the issues caused by the limited receptive field of CNN. Although their trackers work well in regular scenarios, they simply flatten the 2D features into a sequence to better match the Transformer. We believe these operations ignore the spatial prior of the target object, which may lead to sub-optimal results only. In addition, many works demonstrate that self-attention is actually a low-pass filter, which is independent of input features or keys/queries. That is to say, it may suppress the high-frequency component of the input features and preserve or even amplify the low-frequency information. To handle these issues, in this paper, we propose a unified Spatial-Frequency Transformer that models the Gaussian spatial Prior and High-frequency emphasis Attention (GPHA) simultaneously. To be specific, Gaussian spatial prior is generated using dual Multi-Layer Perceptrons (MLPs) and injected into the similarity matrix produced by multiplying Query and Key features in self-attention. The output will be fed into a softmax layer and then decomposed into two components, i.e., the direct and high-frequency signal. The low- and high-pass branches are rescaled and combined to achieve all-pass, therefore, the high-frequency features will be protected well in stacked self-attention layers. We further integrate the Spatial-Frequency Transformer into the Siamese tracking framework and propose a novel tracking algorithm termed SFTransT. The cross-scale fusion based SwinTransformer is adopted as the backbone, and also a multi-head cross-attention module is used to boost the interaction between search and template features. The output will be fed into the tracking head for target localization. Extensive experiments on short-term and long-term tracking benchmarks all demonstrate the effectiveness of our proposed framework. Source code will be released at https://github.com/Tchuanm/SFTransT.git.

Sparse Transformer-Based Sequence Generation for Visual Object Tracking

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

Robust Visual Tracking Via CAMShift and Structural Local Sparse Appearance Model

Efficient transformer tracking with adaptive attention

A Transformer-based Network with Adaptive Spatial Prior for Visual Tracking

Adaptive sparse attention-based compact transformer for object tracking

Learning Spatial-Frequency Transformer for Visual Object Tracking

Exploiting More Information in Sparse Point Cloud for 3D Single Object Tracking

Exploiting spatial relationships for visual tracking

Target-aware transformer tracking with hard occlusion instance generation

Transformer-based visual object tracking via fine-coarse concatenated attention and cross concatenated MLP

VideoTrack: Learning to Track Objects Via Video Transformer

Compact Transformer Tracker with Correlative Masked Modeling.

A transformer‐based lightweight method for multiple‐object tracking

Adaptively Bypassing Vision Transformer Blocks for Efficient Visual Tracking

AiATrack: Attention in Attention for Transformer Visual Tracking

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking

Modeling of Multiple Spatial-Temporal Relations for Robust Visual Object Tracking

Learning Transform-Aware Attentive Network for Object Tracking

Visual tracking using transformer with a combination of convolution and attention

Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking