Abstract:Although the red, green, and blue (RGB) image has a high spatial resolution, it only depicts color intensities in RGB channels, which easily leads to the failure of the tracker based on RGB modality in some challenging scenarios, for example, when the color of the object and background is similar. The hyperspectral image with rich spectral information is more robust in these difficult situations, so it is essential to explore how to effectively apply hyperspectral features to supplement RGB information in object tracking. However, there is no fusion tracking algorithm based on hyperspectral and RGB data. Based on this, we propose a novel fusion tracking framework of hyperspectral and RGB in this article, termed as transformer-based fusion tracking network (TFTN), to enhance the performance of object tracking. Within the framework, we construct a dual-branch structure based on the Siamese network to obtain the modality-specific representations of different modality images. Besides, the framework is generic, which is suitable for the Siamese series of tracking algorithms. In addition, we design a Siamese 3-D convolutional neural network as the specific branch of hyperspectral modality for synchronous extraction of the spatial and spectral features of hyperspectral data, to give full play to the role of hyperspectral data in improving network tracking performance. Particularly, inspired by the structure of Transformer, we design a transformer-based fusion module to capture the potential interaction of intramodality and intermodality features of different modalities. This is the first work that combines the information of hyperspectral and RGB modalities to improve tracking performance. At the same time, it is also the first time that employs the self-attention module of Transformer to combine the information of different modalities for multimodality fusion tracking. Experimental results on the dataset composed of hyperspectral and RGB image sequences show that the proposed TFTN tracker is superior to the state-of-the-art trackers, demonstrating the effectiveness of this method.

DHFN: A Dual-branch Network with Hybrid Fusion for Spatial-Aware Hyperspectral Object Tracking

Background-aware Siamese Network Tracking Based on Salient Feature Fusion

SiamHYPER: Learning a Hyperspectral Object Tracker from an RGB-Based Tracker

Dual-Branch Feature Fusion Network for Salient Object Detection

An Anchor-Free Siamese Target Tracking Network for Hyperspectral Video.

Dual-Focal Camera Hdr Imaging Based On Convolutional Neural Network

Hyperspectral Video Tracker Based on Spectral Deviation Reduction and a Double Siamese Network

HHTrack: Hyperspectral Object Tracking Using Hybrid Attention

Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking.

SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking

TFTN: A Transformer-Based Fusion Tracking Framework of Hyperspectral and RGB

Transformer-Based Band Regrouping With Feature Refinement for Hyperspectral Object Tracking

Learning Dual-Fused Modality-Aware Representations for RGBD Tracking

Hyperspectral Attention Network for Object Tracking

A Channel Adaptive Dual Siamese Network for Hyperspectral Object Tracking

Dual Deep Network for Visual Tracking

Hy-Tracker: A Novel Framework for Enhancing Efficiency and Accuracy of Object Tracking in Hyperspectral Videos

Material-Guided Multiview Fusion Network for Hyperspectral Object Tracking

DHFNet: Decoupled Hierarchical Fusion Network for RGB-T dense prediction tasks

DFSTrack: Dual-stream fusion Siamese network for human pose tracking in videos

Attention fusion and target-uncertain detection for discriminative tracking