Abstract:High spatial, high spectral, and high temporal (H³) information of the objects of interest can be provided by hyperspectral video, which makes it possible to track objects in complex scenarios. However, during motion, changes in the target’s appearance, background, and spectral information can degrade the performance of existing hyperspectral trackers due to insufficient training data. Consequently, this results in weak generalization of these trackers. In this paper, to solve the above problems, a deep temporal-spectral-spatial anchor-free Siamese tracking network for hyperspectral video object tracking, namely HA-Net, is proposed. In HA-Net, a Siamese spectral enhancement tracker module based on an RGB tracker (pseudo-color tracker) is designed, which uses the powerful feature expression capabilities of the deep network to learn more discriminative deep spectral features for identifying objects in complex scenarios. The pseudo-color tracker is introduced to solve the problem of model performance limitation due to insufficient training data. By introducing the temporal-spectral-spatial online discrimination learning module, the temporal-spectral-spatial information of the target can be dynamically modeled to adapt to new targets and the dynamic changes of targets. Benefiting from the double Siamese network architecture, the model can be effectively trained from scratch with less than 20,000 training samples. Online learning of temporal-spectral-spatial information for the target, particularly in cases of insufficient training data, can alleviate the issue of model degradation. This approach enhances the model’s robustness when tracking the target in complex scenes. In the 2021 IEEE WHISPERS Hyperspectral Object Tracking (HOT) Challenge, HA-Net obtained the best performance, with a distance precision (DP) score of 0.948 and an area under the curve (AUC) score of 0.688. The running speed is also 14 frames per second, which is superior to the existing hyperspectral object trackers for hyperspectral video. The source code is available at: https://github.com/zhenliuzhenqi/HOT.

An Anchor-Free Siamese Target Tracking Network for Hyperspectral Video.

A Deep Temporal-Spectral-Spatial Anchor-Free Siamese Tracking Network for Hyperspectral Video Object Tracking

SiamHYPER: Learning a Hyperspectral Object Tracker from an RGB-Based Tracker

Unsupervised Deep Hyperspectral Video Target Tracking and High Spectral-Spatial-Temporal Resolution (H³ Benchmark Dataset

Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking.

SiamOHOT: A Lightweight Dual Siamese Network for Onboard Hyperspectral Object Tracking Via Joint Spatial-Spectral Knowledge Distillation

Background-aware Siamese Network Tracking Based on Salient Feature Fusion

Spectral-Spatial-Aware Transformer Fusion Network For Hyperspectral Object Tracking

A Siamese Network-Based Tracking Framework for Hyperspectral Video

Siammraan: Siamese Multi-Level Residual Attention Adaptive Network for Hyperspectral Videos Tracking.

Learning a Deep Ensemble Network with Band Importance for Hyperspectral Object Tracking.

Hyperspectral Video Tracker Based on Spectral Deviation Reduction and a Double Siamese Network

SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking

MHASiam: Mixed High-Order Attention Siamese Network for Real-Time Visual Tracking

Bae-Net: A Band Attention Aware Ensemble Network For Hyperspectral Object Tracking

Hyperspectral Attention Network for Object Tracking

A Spectral–Spatial Transformer Fusion Method for Hyperspectral Video Tracking

Siamese anchor-free object tracking with multiscale spatial attentions

HSPTrack: Hyperspectral Sequence Prediction Tracker with Transformers

Siamese Network with Multi-Scale Fusion Attention for Visual Tracking

A Channel Adaptive Dual Siamese Network for Hyperspectral Object Tracking