Abstract:Hyperspectral videos can provide the spatial, spectral, and motion information of targets, which makes it possible to track camouflaged targets that are similar to the background. However, hyperspectral object tracking is a challenging task, due to the huge hyperspectral video data dimension and the “data hungry” problem for the model training. Insufficient training data can seriously interfere with the accuracy and generalization of the tracking models. In this paper, a dual deep Siamese network framework for hyperspectral object tracking (SiamHYPER) is proposed for learning a hyperspectral tracker from a pretrained RGB tracker in the case of the “data hungry” problem. Specifically, in addition to a pretrained RGB-based Siamese tracker, a hyperspectral target-aware module is designed to mine the spectral information during the target prediction, and a spatial-spectral cross-attention module is introduced to further fuse the deep spatial and spectral features extracted from the RGB tracker and the hyperspectral target-aware module. Benefiting from the guidance training of the RGB tracker, a robust hyperspectral object tracker can be trained effectively with only a small number of hyperspectral video samples, to overcome the “data hungry” problem. In the experiments conducted in this study, the SiamHYPER framework was verified using SiamBAN and SiamRPN++, with 13 000 frames of hyperspectral videos for training, and achieved the best performance on the publicly available hyperspectral dataset released as part of the WHISPERS Hyperspectral Object Tracking Challenge. The area under the curve (AUC) of SiamHYPER was increased by nearly 8.9% and 7.2%, respectively, when compared with the current state-of-the-art RGB-based and hyperspectral trackers. In addition, the processing speed of SiamHYPER was 19 FPS, which is much higher than that of the current state-of-the-art hyperspectral trackers. The source code is available at zhenliuzhenqi/HOT: Hyperspectral object tracking (github.com).

DSP-Net: A Dynamic Spectral–Spatial Joint Perception Network for Hyperspectral Target Tracking

Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking.

SiamHYPER: Learning a Hyperspectral Object Tracker from an RGB-Based Tracker

An Anchor-Free Siamese Target Tracking Network for Hyperspectral Video.

SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking

Unsupervised Deep Hyperspectral Video Target Tracking and High Spectral-Spatial-Temporal Resolution (H³ Benchmark Dataset

Spnet: A Spectral Patching Network For End-To-End Hyperspectral Image Classification

Background-aware Siamese Network Tracking Based on Salient Feature Fusion

Learning a Deep Ensemble Network with Band Importance for Hyperspectral Object Tracking.

SiamOHOT: A Lightweight Dual Siamese Network for Onboard Hyperspectral Object Tracking Via Joint Spatial-Spectral Knowledge Distillation

Hyperspectral Video Target Tracking based on Pixel-wise Spectral Matching Reduction and Deep Spectral Cascading Texture Features

Hyperspectral Video Tracker Based on Spectral Deviation Reduction and a Double Siamese Network

SSTtrack: A Unified Hyperspectral Video Tracking Framework via Modeling Spectral-Spatial-Temporal Conditions

Hyperspectral Attention Network for Object Tracking

A Channel Adaptive Dual Siamese Network for Hyperspectral Object Tracking

Object Tracking in Hyperspectral-Oriented Video with Fast Spatial-Spectral Features

Bae-Net: A Band Attention Aware Ensemble Network For Hyperspectral Object Tracking

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

Learning Multidimensional Spatial Attention for Robust Nighttime Visual Tracking

Study on the Moving Target Tracking Based on Vision DSP

Deep Spatial-Spectral Joint-Sparse Prior Encoding Network for Hyperspectral Target Detection