Abstract:High spatial, high spectral, and high temporal (H³) information of the objects of interest can be provided by hyperspectral video, which makes it possible to track objects in complex scenarios. However, during motion, changes in the target’s appearance, background, and spectral information can degrade the performance of existing hyperspectral trackers due to insufficient training data. Consequently, this results in weak generalization of these trackers. In this paper, to solve the above problems, a deep temporal-spectral-spatial anchor-free Siamese tracking network for hyperspectral video object tracking, namely HA-Net, is proposed. In HA-Net, a Siamese spectral enhancement tracker module based on an RGB tracker (pseudo-color tracker) is designed, which uses the powerful feature expression capabilities of the deep network to learn more discriminative deep spectral features for identifying objects in complex scenarios. The pseudo-color tracker is introduced to solve the problem of model performance limitation due to insufficient training data. By introducing the temporal-spectral-spatial online discrimination learning module, the temporal-spectral-spatial information of the target can be dynamically modeled to adapt to new targets and the dynamic changes of targets. Benefiting from the double Siamese network architecture, the model can be effectively trained from scratch with less than 20,000 training samples. Online learning of temporal-spectral-spatial information for the target, particularly in cases of insufficient training data, can alleviate the issue of model degradation. This approach enhances the model’s robustness when tracking the target in complex scenes. In the 2021 IEEE WHISPERS Hyperspectral Object Tracking (HOT) Challenge, HA-Net obtained the best performance, with a distance precision (DP) score of 0.948 and an area under the curve (AUC) score of 0.688. The running speed is also 14 frames per second, which is superior to the existing hyperspectral object trackers for hyperspectral video. The source code is available at: https://github.com/zhenliuzhenqi/HOT.

A Channel Adaptive Dual Siamese Network for Hyperspectral Object Tracking

SiamHYPER: Learning a Hyperspectral Object Tracker from an RGB-Based Tracker

SiamOHOT: A Lightweight Dual Siamese Network for Onboard Hyperspectral Object Tracking Via Joint Spatial-Spectral Knowledge Distillation

An Anchor-Free Siamese Target Tracking Network for Hyperspectral Video.

SiamDA: Dual Attention Siamese Network for Real-Time Visual Tracking

Background-aware Siamese Network Tracking Based on Salient Feature Fusion

A Deep Temporal-Spectral-Spatial Anchor-Free Siamese Tracking Network for Hyperspectral Video Object Tracking

Spatial–Spectral Cross-Correlation Embedded Dual-Transfer Network for Object Tracking Using Hyperspectral Videos

Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking.

A Siamese Network-Based Tracking Framework for Hyperspectral Video

Hyperspectral Video Tracker Based on Spectral Deviation Reduction and a Double Siamese Network

Spectral-Spatial-Aware Transformer Fusion Network For Hyperspectral Object Tracking

MHASiam: Mixed High-Order Attention Siamese Network for Real-Time Visual Tracking

Multi-Band Hyperspectral Object Tracking: Leveraging Spectral Information Prompts and Spectral Scale-Aware Representation

Siammraan: Siamese Multi-Level Residual Attention Adaptive Network for Hyperspectral Videos Tracking.

SiamBAG: Band Attention Grouping-Based Siamese Object Tracking Network for Hyperspectral Videos.

Domain Adaptation-aware Transformer for Hyperspectral Object Tracking

SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking

Dual Siamese Network for RGBT Tracking Via Fusing Predicted Position Maps

A Transformer-Based Three-Branch Siamese Network for Hyperspectral Object Tracking

DHFN: A Dual-branch Network with Hybrid Fusion for Spatial-Aware Hyperspectral Object Tracking