Abstract:Visual object tracking often faces challenges such as invalid targets and decreased performance in low-light conditions when relying solely on RGB image sequences. While incorporating additional modalities like depth and infrared data has proven effective, existing multimodal imaging platforms are complex and lack real-world applicability. In contrast, near-infrared (NIR) imaging, commonly used in surveillance cameras, can switch between RGB and NIR based on light intensity. However, tracking objects across these heterogeneous modalities poses significant challenges, particularly due to the absence of modality switch signals during tracking. To address these challenges, we propose an adaptive cross-modal object tracking algorithm called modality-aware fusion network (MAFNet). MAFNet efficiently integrates information from both RGB and NIR modalities using an adaptive weighting mechanism, effectively bridging the appearance gap and enabling a modality-aware target representation. It consists of two key components: an adaptive weighting module and a modality-specific representation module. The adaptive weighting module predicts fusion weights to dynamically adjust the contribution of each modality, while the modality-specific representation module captures discriminative features specific to RGB and NIR modalities. MAFNet offers great flexibility as it can effortlessly integrate into diverse tracking frameworks. With its simplicity, effectiveness, and efficiency, MAFNet outperforms state-of-the-art methods in cross-modal object tracking. To validate the effectiveness of our algorithm and overcome the scarcity of data in this field, we introduce CMOTB, a comprehensive and extensive benchmark dataset for cross-modal object tracking. CMOTB consists of 61 categories and 1000 video sequences, comprising a total of over 799K frames. We believe that our proposed method and dataset offer a strong foundation for advancing cross-modal object-tracking research. The dataset, toolkit, experimental data, and source code will be publicly available at: https://github.com/mmic-lcl/ Datasets-and-benchmark-code.

Reversible Modal Conversion Model for Thermal Infrared Tracking

Modality-transfer Generative Adversarial Network and Dual-Level Unified Latent Representation for Visible Thermal Person Re-Identification

Unsupervised Cross-Modal Distillation for Thermal Infrared Tracking

RGB-T tracking by modality difference reduction and feature re-selection

Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation

Efficient thermal infrared tracking with cross-modal compress distillation

Multi-Level Fusion for Robust RGBT Tracking via Enhanced Thermal Representation

Learning Dual-Level Deep Representation for Thermal Infrared Tracking

Cross-Modal Object Tracking: Modality-Aware Representations and a Unified Benchmark

Visible and Infrared Object Tracking Based on Multimodal Hierarchical Relationship Modeling

Object fusion tracking for RGB-T images via channel swapping and modal mutual attention

Learning a Multimodal Feature Transformer for RGBT Tracking

Transformer-based RGB-T Tracking with Channel and Spatial Feature Fusion

Learning Modality Complementary Features with Mixed Attention Mechanism for RGB-T Tracking

Cross-Modal Object Tracking via Modality-Aware Fusion Network and a Large-Scale Dataset

Visible and Infrared Object Tracking via Convolution-Transformer Network With Joint Multimodal Feature Learning

Real-Time RGBT Target Tracking Based on Attention Mechanism

Synthetic Data Generation for End-to-End Thermal Infrared Tracking

MIRNet: A Robust RGBT Tracking Jointly with Multi-Modal Interaction and Refinement

TIRDet: Mono-Modality Thermal InfraRed Object Detection Based on Prior Thermal-To-Visible Translation

RGB-IR Person Re-identification by Cross-Modality Similarity Preservation