Abstract:RGBT object tracking is an important research topic due to the complementary features of visible and thermal infrared images. However, the established fast-tracking Siamese series RGBT trackers do not fully exploit the predicted results and global information, which leads to poor tracking performance in challenging scenarios such as target object deformation and fast movement. To address these challenges, we propose a dynamic feature-memory transformer RGBT tracking framework in this article. Precisely, we extract the features of visible and infrared image pairs separately using convolutional networks. We then fuse the complementary semantic information using our proposed complementary semantic fusion module, which consists of a group convolution, a channel attention structure, a flattening operation, and channel reduction. Next, we concatenate the dynamic memory feature extracted from the predicted frame pair, the template pair feature, and the search region pair feature. Then input concatenated features into transformer encoder-decoder modules to acquire the global information and further fuse the RGBT features. Finally, we predict the object state using the tracking head while deciding whether to update the dynamic memory feature using the reliability estimator. We conducted extensive experiments on the widely used RGBT234 and LasHeR datasets to evaluate the proposed framework’s performance. The experimental results show that our method outperforms state-of-the-art RGBT trackers, demonstrating the effectiveness of our approach. Overall, our proposed framework leverages visible and infrared images and incorporates global information and dynamic memory features, offering a promising solution to the challenges in RGBT object tracking. Code will be open sourced at https://github.com/ELOESZHANG/DFMTNet

Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation

Unidirectional Cross-Modal Fusion for RGB-T Tracking

Exploring fusion strategies for accurate RGBT visual object tracking

Learning a Multimodal Feature Transformer for RGBT Tracking

Thermal Infrared and Visible Sequences Tracking Via Dual Adversarial Pixel Fusion

Object fusion tracking for RGB-T images via channel swapping and modal mutual attention

RGB-T tracking by modality difference reduction and feature re-selection

RGBT Tracking Based on Modality Feature Enhancement

Unsupervised RGB-T object tracking with attentional multi-modal feature fusion

Multi-scale Feature Extraction and Fusion with Attention Interaction for RGB-T Tracking

Multi-object Tracking Based on Thermal-Visible Video Sequence Fusion

RGB-T Tracking Based on Mixed Attention

Learning Dual-Level Deep Representation for Thermal Infrared Tracking

Robust RGB-T Tracking Via Adaptive Modality Weight Correlation Filters and Cross-modality Learning

Visible–Infrared Dual-Sensor Tracking Based on Transformer via Progressive Feature Enhancement and Fusion

RMFNet: Redetection Multimodal Fusion Network for RGBT Tracking

Dynamic Feature-Memory Transformer Network for RGBT Tracking

Multi-Scale Feature Interactive Fusion Network for RGBT Tracking

Dynamic Fusion Network for RGBT Tracking

Multi-Task Driven Feature Models for Thermal Infrared Tracking.

Simplifying Cross-modal Interaction Via Modality-Shared Features for RGBT Tracking