Abstract:Visual object tracking often faces challenges such as invalid targets and decreased performance in low-light conditions when relying solely on RGB image sequences. While incorporating additional modalities like depth and infrared data has proven effective, existing multimodal imaging platforms are complex and lack real-world applicability. In contrast, near-infrared (NIR) imaging, commonly used in surveillance cameras, can switch between RGB and NIR based on light intensity. However, tracking objects across these heterogeneous modalities poses significant challenges, particularly due to the absence of modality switch signals during tracking. To address these challenges, we propose an adaptive cross-modal object tracking algorithm called modality-aware fusion network (MAFNet). MAFNet efficiently integrates information from both RGB and NIR modalities using an adaptive weighting mechanism, effectively bridging the appearance gap and enabling a modality-aware target representation. It consists of two key components: an adaptive weighting module and a modality-specific representation module. The adaptive weighting module predicts fusion weights to dynamically adjust the contribution of each modality, while the modality-specific representation module captures discriminative features specific to RGB and NIR modalities. MAFNet offers great flexibility as it can effortlessly integrate into diverse tracking frameworks. With its simplicity, effectiveness, and efficiency, MAFNet outperforms state-of-the-art methods in cross-modal object tracking. To validate the effectiveness of our algorithm and overcome the scarcity of data in this field, we introduce CMOTB, a comprehensive and extensive benchmark dataset for cross-modal object tracking. CMOTB consists of 61 categories and 1000 video sequences, comprising a total of over 799K frames. We believe that our proposed method and dataset offer a strong foundation for advancing cross-modal object-tracking research. The dataset, toolkit, experimental data, and source code will be publicly available at: https://github.com/mmic-lcl/ Datasets-and-benchmark-code.

MATI: Multimodal Adaptive Tracking Integrator for Robust Visual Object Tracking

MAT: Motion-Aware Multi-Object Tracking

Multi-features Guided Robust Visual Tracking.

Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusion

SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking

Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking

Transformer-Based Dual-Modal Visual Target Tracking Using Visible Light and Thermal Infrared.

Object Tracking with Adaptive Multicue Incremental Visual Tracker

MM-Tracker: Visual Tracking with A Multi-Task Model Integrating Detection and Differentiating Feature Extraction

Visual Object Tracking With Mutual Affinity Aligned to Human Intuition

MMF-Track: Multi-modal Multi-level Fusion for 3D Single Object Tracking

MCTracker: Satellite Video Multi-Object Tracking Considering Inter-Frame Motion Correlation and Multi-Scale Cascaded Feature Enhancement

Cross-Modal Object Tracking via Modality-Aware Fusion Network and a Large-Scale Dataset

SSTtrack: A Unified Hyperspectral Video Tracking Framework via Modeling Spectral-Spatial-Temporal Conditions

Dual Integration of Multi‐model with Spatial‐temporal Occlusion‐awareness for Visual Object Tracking

Robust Thermal Infrared Tracking Via an Adaptively Multi-Feature Fusion Model.

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

Multi-invariance appearance model for object tracking

VisuaI Tracking Using Adaptive Structure ModeI Based on Spatio-TemporaI Context

Deep Object Tracking with Multi-Modal Data

Robust Object Tracking Based on Timed Motion History Image with Multi-Feature Adaptive Fusion.