Abstract:Hyperspectral video with spatial and spectral information has great potential to improve object tracking performance. However, the limited hyperspectral training samples hinder the development of hyperspectral object tracking. Since hyperspectral data has multiple bands, from which any three bands can be extracted to form pseudocolor images, we propose a Transformer-based multimodality information transfer network (TMTNet), aiming to improve the tracking performance by efficiently transferring the information of multimodality data composed of RGB and hyperspectral in the hyperspectral tracking process. The multimodality information needed to be transferred mainly includes the RGB and hyperspectral multimodality fusion information and the RGB modality information. Specifically, we construct two subnetworks to transfer the multimodality fusion information and the robust RGB visual information, respectively. Among them, the multimodality fusion information transfer subnetwork is designed based on the dual Siamese branch structure. The subnetwork employs the pretrained RGB tracking model as the RGB branch to guide the training of the hyperspectral branch with little training samples. The RGB modality information transfer subnetwork is designed based on a pretrained RGB tracking model with good performance to improve the tracking network’s generalization and accuracy in unknown complex scenes. In addition, we design an information interaction module based on Transformer in the multimodality fusion information transfer subnetwork. The module can fuse multimodality information by capturing the potential interaction between different modalities. We also add a spatial optimization module to TMTNet, which further optimizes the object position predicted by the subject network by fully retaining and utilizing detailed spatial information. Experimental results on the only available hyperspectral tracking benchmark dataset show that the proposed TMTNet tracker outperforms the advanced trackers, demonstrating the effectiveness of this method.

Sparse Mixed Attention Aggregation Network for Multimodal Images Fusion Tracking

RGBT Image Fusion Tracking via Sparse Trifurcate Transformer Aggregation Network

Siamese Network with Multi-Scale Fusion Attention for Visual Tracking

Spectral-Spatial-Aware Transformer Fusion Network For Hyperspectral Object Tracking

CMAT: Integrating Convolution Mixer and Self-Attention for Visual Tracking

An attention-guided matching association network for hyperspectral and rgb fusion tracking

Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusion

Multi-scale Feature Extraction and Fusion with Attention Interaction for RGB-T Tracking

Material-Guided Multiview Fusion Network for Hyperspectral Object Tracking

Mask Guided Spatial-Temporal Fusion Network for Multiple Object Tracking.

A Fusion Approach to Grayscale-Thermal Tracking with Cross-Modal Sparse Representation.

Visible and Infrared Object Tracking via Convolution-Transformer Network With Joint Multimodal Feature Learning

Multi-Scale Feature Interactive Fusion Network for RGBT Tracking

Unsupervised RGB-T object tracking with attentional multi-modal feature fusion

Attention fusion and target-uncertain detection for discriminative tracking

TMTNet: A Transformer-Based Multimodality Information Transfer Network for Hyperspectral Object Tracking.

Learning Collaborative Sparse Representation for Grayscale-Thermal Tracking.

Learning Multi-Layer Attention Aggregation Siamese Network for Robust RGBT Tracking

Robust Thermal Infrared Tracking Via an Adaptively Multi-Feature Fusion Model.

A Robust Attention-Enhanced Network with Transformer for Visual Tracking.

MMF-Track: Multi-modal Multi-level Fusion for 3D Single Object Tracking