Abstract:Hyperspectral video with spatial and spectral information has great potential to improve object tracking performance. However, the limited hyperspectral training samples hinder the development of hyperspectral object tracking. Since hyperspectral data has multiple bands, from which any three bands can be extracted to form pseudocolor images, we propose a Transformer-based multimodality information transfer network (TMTNet), aiming to improve the tracking performance by efficiently transferring the information of multimodality data composed of RGB and hyperspectral in the hyperspectral tracking process. The multimodality information needed to be transferred mainly includes the RGB and hyperspectral multimodality fusion information and the RGB modality information. Specifically, we construct two subnetworks to transfer the multimodality fusion information and the robust RGB visual information, respectively. Among them, the multimodality fusion information transfer subnetwork is designed based on the dual Siamese branch structure. The subnetwork employs the pretrained RGB tracking model as the RGB branch to guide the training of the hyperspectral branch with little training samples. The RGB modality information transfer subnetwork is designed based on a pretrained RGB tracking model with good performance to improve the tracking network’s generalization and accuracy in unknown complex scenes. In addition, we design an information interaction module based on Transformer in the multimodality fusion information transfer subnetwork. The module can fuse multimodality information by capturing the potential interaction between different modalities. We also add a spatial optimization module to TMTNet, which further optimizes the object position predicted by the subject network by fully retaining and utilizing detailed spatial information. Experimental results on the only available hyperspectral tracking benchmark dataset show that the proposed TMTNet tracker outperforms the advanced trackers, demonstrating the effectiveness of this method.

MTNet: Learning Modality-aware Representation with Transformer for RGBT Tracking

Learning Modality Feature Fusion Via Transformer for RGBT-tracking

Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens

Learning Reliable Modal Weight with Transformer for Robust RGBT Tracking

Unified Single-Stage Transformer Network for Efficient RGB-T Tracking

TMTNet: A Transformer-Based Multimodality Information Transfer Network for Hyperspectral Object Tracking.

Highly Compact Adaptive Network Based on Transformer for RGBT Tracking

Transformer-based RGB-T Tracking with Channel and Spatial Feature Fusion

Learning a Multimodal Feature Transformer for RGBT Tracking

Exploring Multi-Modal Spatial-Temporal Contexts for High-Performance RGB-T Tracking

Visible and Infrared Object Tracking Based on Multimodal Hierarchical Relationship Modeling

RGBT Image Fusion Tracking via Sparse Trifurcate Transformer Aggregation Network

MIRNet: A Robust RGBT Tracking Jointly with Multi-Modal Interaction and Refinement

Visible and Infrared Object Tracking via Convolution-Transformer Network With Joint Multimodal Feature Learning

RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learning

Siamese transformer RGBT tracking

Dynamic Feature-Memory Transformer Network for RGBT Tracking

Learning Modality Complementary Features with Mixed Attention Mechanism for RGB-T Tracking

X Modality Assisting RGBT Object Tracking

CMC2R: Cross‐modal Collaborative Contextual Representation for RGBT Tracking

Unidirectional Cross-Modal Fusion for RGB-T Tracking